TensorFlow is an open source machine learning framework. It is basically a library for numerical calculations based on data stream graphs. The graph nodes represent mathematical operations, while the edges of the graph represent multidimensional data arrays (tensors) flowing between them.
Photo: Andrey_Popov – shutterstock.com
This flexible architecture makes it possible to provide calculations on one or more CPUs or GPUs in a desktop PC, server or mobile device without having to rewrite code. TensorFlow also includes TensorBoard, a Tool for data visualization.
While TensorFlow can still build data flow graphs and later run in sessions, version 2.0 fully supports it Eager execution mode. It is an imperative define-by-run interface that instantly calculates operations without building a graph. The eager execution mode supports automatic differentiation via the tf.GradientTape API. One of the improvements in tf.keras (see Keras below) is support for Eager Execution.
While some other APIs have been dropped in TensorFlow 2.0, one is Conversion tool for existing code, in addition to a compatibility library.
Estimators are the most scalable and production-oriented model types for machine learning from TensorFlow. Companies can either use the pre-made Estimators use from Google or write your own. Estimators themselves are built on tf.keras.layers, which simplifies customization. It is usually more convenient to build models with estimators rather than using TensorFlow’s low-level APIs. Pre-built estimators make it possible to work at a higher conceptual level than with the basic TensorFlow APIs
TensorFlow still supports its original low-level API. However, tf.keras has now become the preferred one High level API, an implementation of the Keras API standard that includes improvements specific to TensorFlow. “High” and “Low” refer to how deep and “hardware-related” the API applies. Low level means that more detailed but also more complex settings can be made. The functions are abstracted at the high level, so that although fewer options are available, the API is easier to use.
Keras is a high-level API for neural networks. It is written in Python and can run on top of TensorFlow, CNTK, or Theano. Additional backends such as MXNet or PlaidML are supported by third parties.
Keras was developed with user friendliness in mind. It should be modular, easy to extend, and written in Python. The API was developed for “people and not machines” and follows best practices that are intended to reduce the cognitive load during operation.
Neural layers, cost functions, optimizers, initialization schemes, activation functions and regulation schemes are all independent modules in Keras. They can be combined to create new models. New models can be easily added as new classes and functions. Models are defined in Python code rather than as separate model configuration files.
The main reasons for the use of Keras lie in the design principles followed, above all the focus on user friendliness. It is easy to learn and models are easy to build. In addition, Keras offers the benefits of a large user base, supports a wide range of deployment options, multiple GPUs and distributed training. Google, Microsoft, Amazon, Apple, Nvidia, Uber and many others support the tool.
The Optimization tool Pruning API from Google is technically based on Keras. Therefore, it should be easy to integrate into existing Keras projects. The tool is intended to optimize machine learning models already in the training phase.
According to the name, the API trims ML models. During training, she evaluates the connections between the different levels of the model. Unimportant or irrelevant connections are removed from the network. This reduces both the required memory capacity when saving the model, the working memory required to execute the model and the required CPU operations.
Horovod is one developed by Uber distributed training framework for TensorFlow, Keras and the open source program library PyTorch. Horovod should make distributed deep learning quick and easy to use. It is based on ideas from Baidu’s experimental implementation of the TensorFlow Ring Allreduce algorithm.
Uber originally tried to use Distributed TensorFlow with parameter servers. The engineers found that the message passing interface (MPI) model was less complicated and needed less code adjustments. Uber claims that the Horovod system enables an AI model to be trained about twice as fast as a traditional TensorFlow implementation.
Horovod uses Open MPI (or another MPI implementation) to exchange messages between nodes, and Nvidia NCCL for its highly optimized version of Ring Allreduce. Horovod achieves 90 percent scaling efficiency for Inception-v3 and ResNet-101. For VGG-16, it achieves a scaling efficiency of 68 percent on up to 512 Nvidia Pascal GPUs.
In December 2018, Uber announced that it was sponsoring the Horovod project under the aegis of the LF Deep Learning Foundation for Linux Foundation open source AI software.
LinkedIn made the code for their Tony project public in late 2018. According to Serdar Yegulalp from InfoWorld, the open source tool is used to Manage deep learning jobs in TensorFlow and scale. It uses the YARN (Yet Another Resource Negotiator) job planning system in Hadoop.
While there are already some other planning tools, LinkedIn found a few limitations. TensorFlow on Spark, for example, operates the framework on the Apache Spark job engine, but is therefore very closely linked to Spark. TensorFlowOnYARN offers the same basic features as Tony, but is not maintained and does not offer fault tolerance.
According to LinkedIn, Tony uses the YARN resource and task planning system to set up TensorFlow jobs in a Hadoop cluster. The tool should also make it possible
-
Schedule GPU-based TensorFlow jobs through Hadoop;
-
request different types of resources (CPUs or GPUs);
-
Allocate memory differently for TensorFlow nodes;
-
Store job results regularly in the Hadoop Distributed File System (HDFS) and resume at one point if they are interrupted or crash.
Tony divides the work into three different internal components: a client, an application master, and one that does the job. The client receives incoming TensorFlow jobs. The master coordinates with YARN’s resource manager how the job should be provisioned in YARN. The executing component is what is actually running on the YARN cluster to process the TensorFlow job.
According to LinkedIn, Tony has no noticeable overhead for TensorFlow because it is in the layer that orchestrates distributed TensorFlow. Therefore, it does not affect the actual execution of the TensorFlow job.
Tony also works with the TensorBoard app to visualize TensorFlow, optimize it and free it from errors (debugging).
Inkling is
a commercial one High level programming language from Bonsai (now a Microsoft subsidiary), which makes it easier to build AI applications. According to Paul Krill from InfoWorld, it compiles down to the TensorFlow library. Inkling is designed to present AI in a way that programmers can focus on teaching a system something, rather than focusing on the low-level mechanics.
Inkling abstracts dynamic AI algorithms that normally require specialist knowledge in the field of machine learning. According to Bonsai, the language is descriptive and the syntax is reminiscent of a mixture of Python and SQL. The aim of the language is to make machine learning accessible to developers and engineers who have no ML background but want to use the technology in their respective specialist areas.
Google’s open source project Tensor2Tensor (T2T) should Reduce workloadto configure a deep learning model for training. It is a Python-based library for workflow optimization of TensorFlow training tasks. This allows developers to specify the key elements in a TensorFlow model and define their relationships with one another.
The key elements according to InfoWorld editor Serdar Yegulalp are:
-
Records: T2T already supports numerous data sets for the training. New data records can be added to the individual workflow or via pull request to the core T2T project.
-
Problems and modalities: These describe which task is being trained for (e.g. speech recognition or translation) and which data should both be received and generated from it. For example, an image recognition system would receive image data and output text descriptions.
-
Models: Many common models are already registered with T2T and more can be added.
-
Hyperparameters: A hyper parameter is a parameter that is used to control the ML training algorithm and, unlike other parameters, its value is not learned in the actual training of the model. Different sets of settings that control the training process can be created in T2T. These can be changed or interconnected as required.
-
Trainer: The parameters that are transferred to the actual training binary file can be specified separately.
T2T provides presets for each element. Numerous common models and data sets are already included. This allows training to begin quickly by reusing or expanding it. What T2T doesn’t deliver is a broader context beyond TensorFlow, how the deep learning project should be organized. It “only” makes it easier to use the framework.
Another tool that that Simplify deep learning training should come as open source from the Horovod makers Uber. With Ludwig, ML models should be able to be trained and tested without any programming effort.
According to Heise, the tool takes a data type-related approach to model development, with specific encoders and decoders for each data type. Different encoders and decoders should also be used for each type. The desired encoder and hyper parameters should be able to be set directly in the model configuration file (in YAML format) without having to write code. In the current version, Ludwig offers encoders and decoders for common data types such as binary values, floating point numbers, categories, discrete sequences, images, texts and time series, which should be supplemented by pre-trained models if necessary.
With TensorWatch, Microsoft Research released an open sourceDebugging tool for machine learning, which should also help with complex problems.
Often, no log files are created during model training because, depending on the size of the data records, they can cause high storage costs. However, this means that there is no overall overview of errors in the model.
TensorWatch is to visualize interactive real-time debugging in Jupyter Notebook as well as offer custom UIs and the possibility of integration into the Python ecosystem. Jupyter Notebook is an open source web application that can be used to create and share documents that contain live code, formulas, visualizations or text. Microsoft provides bar graphs, histograms, pie charts and 3D variations as supported visualization types.
In TensorWatch, data and other objects are generally treated as streams – including files, consoles, sockets, cloud storage and the visualizations themselves. These streams can “listen” to other streams so that, for example, several of them can be combined into one visualization or one stream in several visualizations at the same time.
With the “Lazy Logging Mode” TensorWatch can monitor the variables of the model without explicit logging. Users can make interactive requests in real time that run in the context of these variables and return a stream as a result. These in turn can be visualized, saved or processed.
The open source MorphNet TensorFlow extension from Google is a tool to deep improve neural networksby adapting the network architecture to new requirements. To do this, the tool analyzes the neural networks and then optimizes them by changing their structure.
According to JAXenter, the tool works in two steps:
-
The shrinkage: MorphNet analyzes the structure of a network and the efficiency of the individual neurons based on a criterion to be optimized. Neurons rated as inefficient are removed from the network.
-
The extension: After removing the inefficient neurons, the tool expands all layers of the network by an identical factor. Efficient layers, from which fewer neurons have been removed, increase in size relatively. The expansion is less than the shrinkage, saving resources while increasing efficiency and maintaining accuracy.