Aprendizado de máquina robusto em streaming de dados usando Kafka e Tensorflow-IO

Ver no TensorFlow.org Executar no Google Colab Ver fonte no GitHub Baixar caderno

Visão geral

Este tutorial concentra-se em streaming de dados de um Kafka conjunto em um tf.data.Dataset que é então utilizado em conjunto com tf.keras de formação e inferência.

Kafka é principalmente uma plataforma de streaming de eventos distribuída que fornece dados de streaming escaláveis ​​e tolerantes a falhas em pipelines de dados. É um componente técnico essencial de uma infinidade de grandes empresas onde a entrega de dados de missão crítica é um requisito primário.

Configurar

Instale os pacotes tensorflow-io e kafka necessários

pip install -q tensorflow-io
pip install -q kafka-python

Pacotes de importação

import os
from datetime import datetime
import time
import threading
import json
from kafka import KafkaProducer
from kafka.errors import KafkaError
from sklearn.model_selection import train_test_split
import pandas as pd
import tensorflow as tf
import tensorflow_io as tfio

Validar importações tf e tfio

print("tensorflow-io version: {}".format(tfio.__version__))
print("tensorflow version: {}".format(tf.__version__))
tensorflow-io version: 0.17.0
tensorflow version: 2.4.1

Baixe e configure as instâncias do Kafka e do Zookeeper

Para fins de demonstração, as seguintes instâncias são configuradas localmente:

  • Kafka (Corretores: 127.0.0.1:9092)
  • Zookeeper (Nó: 127.0.0.1:2181)
curl -sSOL https://downloads.apache.org/kafka/2.7.0/kafka_2.13-2.7.0.tgz
tar -xzf kafka_2.13-2.7.0.tgz

Usando as configurações padrão (fornecidas pelo Apache Kafka) para ativar as instâncias.

./kafka_2.13-2.7.0/bin/zookeeper-server-start.sh -daemon ./kafka_2.13-2.7.0/config/zookeeper.properties
./kafka_2.13-2.7.0/bin/kafka-server-start.sh -daemon ./kafka_2.13-2.7.0/config/server.properties
echo "Waiting for 10 secs until kafka and zookeeper services are up and running"
sleep 10
Waiting for 10 secs until kafka and zookeeper services are up and running

Uma vez que as instâncias são iniciadas como processos daemon, grep para kafka na lista de processos. Os dois processos java correspondem às instâncias zookeeper e kafka.

ps -ef | grep kafka
kbuilder  3128 22247  2 19:38 ?        00:00:00 python /tmpfs/src/gfile/executor.py --input_notebook=/tmpfs/src/temp/docs/tutorials/kafka.ipynb --timeout=15000
kbuilder  3562     1 13 19:38 ?        00:00:01 java -Xmx512M -Xms512M -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -XX:MaxInlineLevel=15 -Djava.awt.headless=true -Xlog:gc*:file=/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../logs/zookeeper-gc.log:time,tags:filecount=10,filesize=100M -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../logs -Dlog4j.configuration=file:./kafka_2.13-2.7.0/bin/../config/log4j.properties -cp /tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/activation-1.1.1.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/aopalliance-repackaged-2.6.1.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/argparse4j-0.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/audience-annotations-0.5.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/commons-cli-1.4.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/commons-lang3-3.8.1.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/connect-api-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/connect-basic-auth-extension-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/connect-file-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/connect-json-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/connect-mirror-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/connect-mirror-client-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/connect-runtime-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/connect-transforms-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/hk2-api-2.6.1.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/hk2-locator-2.6.1.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/hk2-utils-2.6.1.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jackson-annotations-2.10.5.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jackson-core-2.10.5.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jackson-databind-2.10.5.1.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jackson-dataformat-csv-2.10.5.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jackson-datatype-jdk8-2.10.5.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jackson-jaxrs-base-2.10.5.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jackson-jaxrs-json-provider-2.10.5.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jackson-module-jaxb-annotations-2.10.5.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jackson-module-paranamer-2.10.5.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jackson-module-scala_2.13-2.10.5.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jakarta.activation-api-1.2.1.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jakarta.annotation-api-1.3.5.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jakarta.inject-2.6.1.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jakarta.validation-api-2.0.2.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jakarta.ws.rs-api-2.1.6.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/javassist-3.25.0-GA.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/javassist-3.26.0-GA.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/javax.servlet-api-3.1.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/javax.ws.rs-api-2.1.1.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jaxb-api-2.3.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jersey-client-2.31.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jersey-common-2.31.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jersey-container-servlet-2.31.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jersey-container-servlet-core-2.31.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jersey-hk2-2.31.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jersey-media-jaxb-2.31.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jersey-server-2.31.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jetty-client-9.4.33.v20201020.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jetty-continuation-9.4.33.v20201020.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jetty-http-9.4.33.v20201020.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jetty-io-9.4.33.v20201020.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jetty-security-9.4.33.v20201020.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jetty-server-9.4.33.v20201020.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jetty-servlet-9.4.33.v20201020.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jetty-servlets-9.4.33.v20201020.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jetty-util-9.4.33.v20201020.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jopt-simple-5.0.4.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/kafka-clients-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/kafka-log4j-appender-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/kafka-raft-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/kafka-streams-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/kafka-streams-examples-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/kafka-streams-scala_2.13-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/kafka-streams-test-utils-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/kafka-tools-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/kafka_2.13-2.7.0-sources.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/kafka_2.13-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/log4j-1.2.17.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/lz4-java-1.7.1.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/maven-artifact-3.6.3.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/metrics-core-2.2.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/netty-buffer-4.1.51.Final.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/netty-codec-4.1.51.Final.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/netty-common-4.1.51.Final.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/netty-handler-4.1.51.Final.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/netty-resolver-4.1.51.Final.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/netty-transport-4.1.51.Final.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/netty-transport-native-epoll-4.1.51.Final.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/netty-transport-native-unix-common-4.1.51.Final.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/osgi-resource-locator-1.0.3.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/paranamer-2.8.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/plexus-utils-3.2.1.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/reflections-0.9.12.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/rocksdbjni-5.18.4.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/scala-collection-compat_2.13-2.2.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/scala-java8-compat_2.13-0.9.1.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/scala-library-2.13.3.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/scala-logging_2.13-3.9.2.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/scala-reflect-2.13.3.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/slf4j-api-1.7.30.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/slf4j-log4j12-1.7.30.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/snappy-java-1.1.7.7.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/zookeeper-3.5.8.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/zookeeper-jute-3.5.8.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/zstd-jni-1.4.5-6.jar org.apache.zookeeper.server.quorum.QuorumPeerMain ./kafka_2.13-2.7.0/config/zookeeper.properties
kbuilder  3930     1 52 19:38 ?        00:00:05 java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -XX:MaxInlineLevel=15 -Djava.awt.headless=true -Xlog:gc*:file=/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../logs/kafkaServer-gc.log:time,tags:filecount=10,filesize=100M -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../logs -Dlog4j.configuration=file:./kafka_2.13-2.7.0/bin/../config/log4j.properties -cp /tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/activation-1.1.1.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/aopalliance-repackaged-2.6.1.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/argparse4j-0.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/audience-annotations-0.5.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/commons-cli-1.4.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/commons-lang3-3.8.1.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/connect-api-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/connect-basic-auth-extension-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/connect-file-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/connect-json-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/connect-mirror-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/connect-mirror-client-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/connect-runtime-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/connect-transforms-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/hk2-api-2.6.1.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/hk2-locator-2.6.1.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/hk2-utils-2.6.1.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jackson-annotations-2.10.5.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jackson-core-2.10.5.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jackson-databind-2.10.5.1.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jackson-dataformat-csv-2.10.5.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jackson-datatype-jdk8-2.10.5.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jackson-jaxrs-base-2.10.5.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jackson-jaxrs-json-provider-2.10.5.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jackson-module-jaxb-annotations-2.10.5.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jackson-module-paranamer-2.10.5.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jackson-module-scala_2.13-2.10.5.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jakarta.activation-api-1.2.1.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jakarta.annotation-api-1.3.5.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jakarta.inject-2.6.1.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jakarta.validation-api-2.0.2.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jakarta.ws.rs-api-2.1.6.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/javassist-3.25.0-GA.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/javassist-3.26.0-GA.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/javax.servlet-api-3.1.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/javax.ws.rs-api-2.1.1.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jaxb-api-2.3.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jersey-client-2.31.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jersey-common-2.31.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jersey-container-servlet-2.31.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jersey-container-servlet-core-2.31.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jersey-hk2-2.31.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jersey-media-jaxb-2.31.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jersey-server-2.31.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jetty-client-9.4.33.v20201020.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jetty-continuation-9.4.33.v20201020.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jetty-http-9.4.33.v20201020.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jetty-io-9.4.33.v20201020.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jetty-security-9.4.33.v20201020.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jetty-server-9.4.33.v20201020.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jetty-servlet-9.4.33.v20201020.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jetty-servlets-9.4.33.v20201020.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jetty-util-9.4.33.v20201020.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/jopt-simple-5.0.4.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/kafka-clients-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/kafka-log4j-appender-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/kafka-raft-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/kafka-streams-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/kafka-streams-examples-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/kafka-streams-scala_2.13-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/kafka-streams-test-utils-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/kafka-tools-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/kafka_2.13-2.7.0-sources.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/kafka_2.13-2.7.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/log4j-1.2.17.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/lz4-java-1.7.1.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/maven-artifact-3.6.3.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/metrics-core-2.2.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/netty-buffer-4.1.51.Final.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/netty-codec-4.1.51.Final.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/netty-common-4.1.51.Final.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/netty-handler-4.1.51.Final.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/netty-resolver-4.1.51.Final.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/netty-transport-4.1.51.Final.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/netty-transport-native-epoll-4.1.51.Final.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/netty-transport-native-unix-common-4.1.51.Final.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/osgi-resource-locator-1.0.3.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/paranamer-2.8.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/plexus-utils-3.2.1.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/reflections-0.9.12.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/rocksdbjni-5.18.4.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/scala-collection-compat_2.13-2.2.0.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/scala-java8-compat_2.13-0.9.1.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/scala-library-2.13.3.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/scala-logging_2.13-3.9.2.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/scala-reflect-2.13.3.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/slf4j-api-1.7.30.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/slf4j-log4j12-1.7.30.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/snappy-java-1.1.7.7.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/zookeeper-3.5.8.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/zookeeper-jute-3.5.8.jar:/tmpfs/src/temp/docs/tutorials/kafka_2.13-2.7.0/bin/../libs/zstd-jni-1.4.5-6.jar kafka.Kafka ./kafka_2.13-2.7.0/config/server.properties
kbuilder  4118  3132  0 19:38 pts/0    00:00:00 /bin/bash -c ps -ef | grep kafka
kbuilder  4120  4118  0 19:38 pts/0    00:00:00 grep kafka

Crie os tópicos kafka com as seguintes especificações:

  • susy-train: partições = 1, fator de replicação = 1
  • teste susy: partições = 2, fator de replicação = 1
./kafka_2.13-2.7.0/bin/kafka-topics.sh --create --bootstrap-server 127.0.0.1:9092 --replication-factor 1 --partitions 1 --topic susy-train
./kafka_2.13-2.7.0/bin/kafka-topics.sh --create --bootstrap-server 127.0.0.1:9092 --replication-factor 1 --partitions 2 --topic susy-test
Created topic susy-train.
Created topic susy-test.

Descreva o tópico para obter detalhes sobre a configuração

./kafka_2.13-2.7.0/bin/kafka-topics.sh --describe --bootstrap-server 127.0.0.1:9092 --topic susy-train
./kafka_2.13-2.7.0/bin/kafka-topics.sh --describe --bootstrap-server 127.0.0.1:9092 --topic susy-test
Topic: susy-train PartitionCount: 1 ReplicationFactor: 1  Configs: segment.bytes=1073741824
    Topic: susy-train Partition: 0  Leader: 0 Replicas: 0   Isr: 0
Topic: susy-test  PartitionCount: 2 ReplicationFactor: 1  Configs: segment.bytes=1073741824
    Topic: susy-test  Partition: 0  Leader: 0 Replicas: 0   Isr: 0
    Topic: susy-test  Partition: 1  Leader: 0 Replicas: 0   Isr: 0

O fator de replicação 1 indica que os dados não estão sendo replicados. Isso se deve à presença de um único corretor em nossa configuração kafka. Em sistemas de produção, o número de servidores de bootstrap pode estar na faixa de 100 nós. É aí que entra em cena a tolerância a falhas usando replicação.

Por favor, consulte os docs para mais detalhes.

Conjunto de dados SUSY

Kafka sendo uma plataforma de streaming de eventos, permite que dados de várias fontes sejam gravados nela. Por exemplo:

  • Registros de tráfego da web
  • Medidas astronômicas
  • Dados do sensor IoT
  • Avaliações de produtos e muito mais.

Para o propósito deste tutorial, vamos baixar o SUSY conjunto de dados e alimentar os dados em kafka manualmente. O objetivo deste problema de classificação é distinguir entre um processo de sinal que produz partículas supersimétricas e um processo de fundo que não o faz.

curl -sSOL https://archive.ics.uci.edu/ml/machine-learning-databases/00279/SUSY.csv.gz

Explore o conjunto de dados

A primeira coluna é o rótulo da classe (1 para sinal, 0 para plano de fundo), seguido pelos 18 recursos (8 recursos de baixo nível e 10 recursos de alto nível). Os primeiros 8 recursos são propriedades cinemáticas medidas pelos detectores de partículas no acelerador. Os últimos 10 recursos são funções dos primeiros 8 recursos. Esses são recursos de alto nível derivados por físicos para ajudar a discriminar entre as duas classes.

COLUMNS = [
          #  labels
           'class',
          #  low-level features
           'lepton_1_pT',
           'lepton_1_eta',
           'lepton_1_phi',
           'lepton_2_pT',
           'lepton_2_eta',
           'lepton_2_phi',
           'missing_energy_magnitude',
           'missing_energy_phi',
          #  high-level derived features
           'MET_rel',
           'axial_MET',
           'M_R',
           'M_TR_2',
           'R',
           'MT2',
           'S_R',
           'M_Delta_R',
           'dPhi_r_b',
           'cos(theta_r1)'
           ]

Todo o conjunto de dados consiste em 5 milhões de linhas. No entanto, para o propósito deste tutorial, vamos considerar apenas uma fração do conjunto de dados (100.000 linhas) para que menos tempo seja gasto na movimentação dos dados e mais tempo na compreensão da funcionalidade da API.

susy_iterator = pd.read_csv('SUSY.csv.gz', header=None, names=COLUMNS, chunksize=100000)
susy_df = next(susy_iterator)
susy_df.head()
# Number of datapoints and columns
len(susy_df), len(susy_df.columns)
(100000, 19)
# Number of datapoints belonging to each class (0: background noise, 1: signal)
len(susy_df[susy_df["class"]==0]), len(susy_df[susy_df["class"]==1])
(54025, 45975)

Divida o conjunto de dados

train_df, test_df = train_test_split(susy_df, test_size=0.4, shuffle=True)
print("Number of training samples: ",len(train_df))
print("Number of testing sample: ",len(test_df))

x_train_df = train_df.drop(["class"], axis=1)
y_train_df = train_df["class"]

x_test_df = test_df.drop(["class"], axis=1)
y_test_df = test_df["class"]

# The labels are set as the kafka message keys so as to store data
# in multiple-partitions. Thus, enabling efficient data retrieval
# using the consumer groups.
x_train = list(filter(None, x_train_df.to_csv(index=False).split("\n")[1:]))
y_train = list(filter(None, y_train_df.to_csv(index=False).split("\n")[1:]))

x_test = list(filter(None, x_test_df.to_csv(index=False).split("\n")[1:]))
y_test = list(filter(None, y_test_df.to_csv(index=False).split("\n")[1:]))
Number of training samples:  60000
Number of testing sample:  40000
NUM_COLUMNS = len(x_train_df.columns)
len(x_train), len(y_train), len(x_test), len(y_test)
(60000, 60000, 40000, 40000)

Armazene os dados de trem e teste no kafka

O armazenamento dos dados no kafka simula um ambiente de recuperação remota contínua de dados para fins de treinamento e inferência.

def error_callback(exc):
    raise Exception('Error while sendig data to kafka: {0}'.format(str(exc)))

def write_to_kafka(topic_name, items):
  count=0
  producer = KafkaProducer(bootstrap_servers=['127.0.0.1:9092'])
  for message, key in items:
    producer.send(topic_name, key=key.encode('utf-8'), value=message.encode('utf-8')).add_errback(error_callback)
    count+=1
  producer.flush()
  print("Wrote {0} messages into topic: {1}".format(count, topic_name))

write_to_kafka("susy-train", zip(x_train, y_train))
write_to_kafka("susy-test", zip(x_test, y_test))
Wrote 60000 messages into topic: susy-train
Wrote 40000 messages into topic: susy-test

Defina o conjunto de dados tfio train

O IODataset classe é utilizada para transmissão de dados de kafka em tensorflow. Os herda da classe de tf.data.Dataset e, portanto, tem todas as funcionalidades úteis do tf.data.Dataset fora da caixa.

def decode_kafka_item(item):
  message = tf.io.decode_csv(item.message, [[0.0] for i in range(NUM_COLUMNS)])
  key = tf.strings.to_number(item.key)
  return (message, key)

BATCH_SIZE=64
SHUFFLE_BUFFER_SIZE=64
train_ds = tfio.IODataset.from_kafka('susy-train', partition=0, offset=0)
train_ds = train_ds.shuffle(buffer_size=SHUFFLE_BUFFER_SIZE)
train_ds = train_ds.map(decode_kafka_item)
train_ds = train_ds.batch(BATCH_SIZE)

Construir e treinar o modelo

# Set the parameters

OPTIMIZER="adam"
LOSS=tf.keras.losses.BinaryCrossentropy(from_logits=True)
METRICS=['accuracy']
EPOCHS=10
# design/build the model
model = tf.keras.Sequential([
  tf.keras.layers.Input(shape=(NUM_COLUMNS,)),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(256, activation='relu'),
  tf.keras.layers.Dropout(0.4),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dropout(0.4),
  tf.keras.layers.Dense(1, activation='sigmoid')
])

print(model.summary())
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense (Dense)                (None, 128)               2432      
_________________________________________________________________
dropout (Dropout)            (None, 128)               0         
_________________________________________________________________
dense_1 (Dense)              (None, 256)               33024     
_________________________________________________________________
dropout_1 (Dropout)          (None, 256)               0         
_________________________________________________________________
dense_2 (Dense)              (None, 128)               32896     
_________________________________________________________________
dropout_2 (Dropout)          (None, 128)               0         
_________________________________________________________________
dense_3 (Dense)              (None, 1)                 129       
=================================================================
Total params: 68,481
Trainable params: 68,481
Non-trainable params: 0
_________________________________________________________________
None
# compile the model
model.compile(optimizer=OPTIMIZER, loss=LOSS, metrics=METRICS)
# fit the model
model.fit(train_ds, epochs=EPOCHS)
Epoch 1/10
938/938 [==============================] - 31s 32ms/step - loss: 0.5218 - accuracy: 0.7379
Epoch 2/10
938/938 [==============================] - 31s 32ms/step - loss: 0.4578 - accuracy: 0.7858
Epoch 3/10
938/938 [==============================] - 31s 32ms/step - loss: 0.4516 - accuracy: 0.7906
Epoch 4/10
938/938 [==============================] - 31s 32ms/step - loss: 0.4492 - accuracy: 0.7908
Epoch 5/10
938/938 [==============================] - 31s 32ms/step - loss: 0.4458 - accuracy: 0.7944
Epoch 6/10
938/938 [==============================] - 31s 32ms/step - loss: 0.4466 - accuracy: 0.7947
Epoch 7/10
938/938 [==============================] - 31s 32ms/step - loss: 0.4439 - accuracy: 0.7939
Epoch 8/10
938/938 [==============================] - 31s 32ms/step - loss: 0.4442 - accuracy: 0.7941
Epoch 9/10
938/938 [==============================] - 31s 32ms/step - loss: 0.4421 - accuracy: 0.7951
Epoch 10/10
938/938 [==============================] - 30s 32ms/step - loss: 0.4423 - accuracy: 0.7962
<tensorflow.python.keras.callbacks.History at 0x7fe8b8405b00>

Como apenas uma fração do conjunto de dados está sendo utilizada, nossa precisão é limitada a ~ 78% durante a fase de treinamento. No entanto, sinta-se à vontade para armazenar dados adicionais no kafka para um melhor desempenho do modelo. Além disso, como o objetivo era apenas demonstrar a funcionalidade dos conjuntos de dados tfio kafka, uma rede neural menor e menos complicada foi usada. No entanto, pode-se aumentar a complexidade do modelo, modificar a estratégia de aprendizagem, ajustar hiperparâmetros etc. para fins de exploração. Para uma abordagem de linha de base, consulte este artigo .

Inferir sobre os dados de teste

Para inferir sobre os dados de teste, aderindo à semântica 'exatamente uma vez', juntamente com tolerância a falhas, o streaming.KafkaGroupIODataset pode ser utilizado.

Defina o conjunto de dados de teste tfio

Os stream_timeout blocos de parâmetros para a duração dada para novos pontos de dados a serem transmitidos para o tópico. Isso elimina a necessidade de criar novos conjuntos de dados se os dados estiverem sendo transmitidos para o tópico de maneira intermitente.

test_ds = tfio.experimental.streaming.KafkaGroupIODataset(
    topics=["susy-test"],
    group_id="testcg",
    servers="127.0.0.1:9092",
    stream_timeout=10000,
    configuration=[
        "session.timeout.ms=7000",
        "max.poll.interval.ms=8000",
        "auto.offset.reset=earliest"
    ],
)

def decode_kafka_test_item(raw_message, raw_key):
  message = tf.io.decode_csv(raw_message, [[0.0] for i in range(NUM_COLUMNS)])
  key = tf.strings.to_number(raw_key)
  return (message, key)

test_ds = test_ds.map(decode_kafka_test_item)
test_ds = test_ds.batch(BATCH_SIZE)

Embora essa classe possa ser usada para fins de treinamento, existem ressalvas que precisam ser abordadas. Uma vez que todas as mensagens são lidas a partir kafka e os últimos deslocamentos são cometidos usando o streaming.KafkaGroupIODataset , o consumidor não reiniciar lendo as mensagens desde o início. Assim, durante o treinamento, é possível treinar apenas para uma única época com os dados fluindo continuamente. Este tipo de funcionalidade tem casos de uso limitados durante a fase de treinamento em que, uma vez que um ponto de dados foi consumido pelo modelo, ele não é mais necessário e pode ser descartado.

No entanto, essa funcionalidade brilha quando se trata de inferência robusta com semântica exatamente uma vez.

avaliar o desempenho nos dados de teste

res = model.evaluate(test_ds)
print("test loss, test acc:", res)
625/625 [==============================] - 13s 21ms/step - loss: 0.4367 - accuracy: 0.7980
test loss, test acc: [0.4366855025291443, 0.7980499863624573]

Como a inferência é baseada na semântica 'exatamente uma vez', a avaliação no conjunto de teste pode ser executada apenas uma vez. Para executar a inferência novamente nos dados de teste, um novo grupo de consumidores deve ser usado.

Acompanhe o lag deslocamento da testcg grupo de consumidores

./kafka_2.13-2.7.0/bin/kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --describe --group testcg
GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                  HOST            CLIENT-ID
testcg          susy-test       0          21592           21592           0               rdkafka-66bc39ef-397c-4190-a17c-69ab519d4f53 /10.142.0.106   rdkafka
testcg          susy-test       1          18408           18408           0               rdkafka-66bc39ef-397c-4190-a17c-69ab519d4f53 /10.142.0.106   rdkafka

Uma vez que as current-offset partidas o log-end-offset para todas as partições, isso indica que o consumidor (s) ter concluído buscar todas as mensagens do tópico kafka.

Aprendizagem online

O paradigma do aprendizado de máquina online é um pouco diferente da forma tradicional / convencional de treinar modelos de aprendizado de máquina. No primeiro caso, o modelo continua a aprender / atualizar incrementalmente seus parâmetros assim que os novos pontos de dados estiverem disponíveis e espera-se que esse processo continue indefinidamente. Isso é diferente das últimas abordagens onde o conjunto de dados é fixo e o modelo itera sobre ele n número de vezes. No aprendizado online, os dados uma vez consumidos pelo modelo podem não estar disponíveis para treinamento novamente.

Ao utilizar o streaming.KafkaBatchIODataset , agora é possível treinar os modelos desta forma. Vamos continuar a usar nosso conjunto de dados SUSY para demonstrar essa funcionalidade.

O conjunto de dados de treinamento tfio para aprendizagem online

O streaming.KafkaBatchIODataset é semelhante ao streaming.KafkaGroupIODataset nele de API. Além disso, recomenda-se utilizar o stream_timeout parâmetro para configurar a duração para que o conjunto de dados irá bloquear novas mensagens antes do tempo limite. No exemplo abaixo, o conjunto de dados é configurada com uma stream_timeout de 10000 milissegundos. Isso implica que, depois que todas as mensagens do tópico forem consumidas, o conjunto de dados aguardará mais 10 segundos antes de atingir o tempo limite e se desconectar do cluster kafka. Se novas mensagens forem transmitidas para o tópico antes de atingir o tempo limite, o consumo de dados e o treinamento do modelo serão retomados para os pontos de dados recém-consumidos. Para bloquear indefinidamente, para defini-la -1 .

online_train_ds = tfio.experimental.streaming.KafkaBatchIODataset(
    topics=["susy-train"],
    group_id="cgonline",
    servers="127.0.0.1:9092",
    stream_timeout=10000, # in milliseconds, to block indefinitely, set it to -1.
    configuration=[
        "session.timeout.ms=7000",
        "max.poll.interval.ms=8000",
        "auto.offset.reset=earliest"
    ],
)

Cada item que os online_train_ds gera é um tf.data.Dataset em si. Assim, todas as transformações padrão podem ser aplicadas normalmente.

def decode_kafka_online_item(raw_message, raw_key):
  message = tf.io.decode_csv(raw_message, [[0.0] for i in range(NUM_COLUMNS)])
  key = tf.strings.to_number(raw_key)
  return (message, key)

for mini_ds in online_train_ds:
  mini_ds = mini_ds.shuffle(buffer_size=32)
  mini_ds = mini_ds.map(decode_kafka_online_item)
  mini_ds = mini_ds.batch(32)
  model.fit(mini_ds, epochs=3)
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4279 - accuracy: 0.8115
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4130 - accuracy: 0.8213
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.3996 - accuracy: 0.8242
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4646 - accuracy: 0.7920
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4415 - accuracy: 0.8047
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4296 - accuracy: 0.8018
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4410 - accuracy: 0.7969
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4223 - accuracy: 0.8174
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4187 - accuracy: 0.8096
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4679 - accuracy: 0.7734
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4292 - accuracy: 0.7979
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4411 - accuracy: 0.7959
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4750 - accuracy: 0.7695
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4460 - accuracy: 0.7959
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4345 - accuracy: 0.8008
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4282 - accuracy: 0.8154
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4086 - accuracy: 0.8184
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.3942 - accuracy: 0.8291
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4265 - accuracy: 0.7852
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4233 - accuracy: 0.7988
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4037 - accuracy: 0.8164
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4463 - accuracy: 0.7988
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4064 - accuracy: 0.8164
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4093 - accuracy: 0.8115
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4652 - accuracy: 0.7930
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4508 - accuracy: 0.7930
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4296 - accuracy: 0.8018
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4671 - accuracy: 0.7812
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4519 - accuracy: 0.7910
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4431 - accuracy: 0.7930
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4905 - accuracy: 0.7637
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4691 - accuracy: 0.7764
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4518 - accuracy: 0.7803
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4379 - accuracy: 0.7930
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4314 - accuracy: 0.8057
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4232 - accuracy: 0.7988
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4892 - accuracy: 0.7539
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4660 - accuracy: 0.7773
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4572 - accuracy: 0.7842
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4474 - accuracy: 0.7871
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4435 - accuracy: 0.7920
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4247 - accuracy: 0.8047
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4306 - accuracy: 0.8115
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4219 - accuracy: 0.8164
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4017 - accuracy: 0.8223
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4662 - accuracy: 0.7881
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4563 - accuracy: 0.7988
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4445 - accuracy: 0.7930
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4437 - accuracy: 0.7910
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4282 - accuracy: 0.7988
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4203 - accuracy: 0.8057
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4227 - accuracy: 0.7998
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4005 - accuracy: 0.8223
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.3905 - accuracy: 0.8184
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4617 - accuracy: 0.7871
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4414 - accuracy: 0.7891
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4320 - accuracy: 0.8018
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4493 - accuracy: 0.7900
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4418 - accuracy: 0.8037
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4265 - accuracy: 0.8096
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4740 - accuracy: 0.7783
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4498 - accuracy: 0.7812
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4473 - accuracy: 0.7910
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4439 - accuracy: 0.7900
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4217 - accuracy: 0.8047
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4126 - accuracy: 0.8135
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4417 - accuracy: 0.7998
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4120 - accuracy: 0.8105
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.3967 - accuracy: 0.8213
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4657 - accuracy: 0.7822
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4437 - accuracy: 0.7998
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4344 - accuracy: 0.8057
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4385 - accuracy: 0.8018
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4314 - accuracy: 0.8096
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4186 - accuracy: 0.8086
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4918 - accuracy: 0.7725
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4719 - accuracy: 0.7725
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4586 - accuracy: 0.7920
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4989 - accuracy: 0.7607
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4835 - accuracy: 0.7676
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4796 - accuracy: 0.7695
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4645 - accuracy: 0.7754
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4505 - accuracy: 0.7812
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4420 - accuracy: 0.7852
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4219 - accuracy: 0.8047
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4029 - accuracy: 0.8135
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.3870 - accuracy: 0.8252
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4627 - accuracy: 0.7871
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4313 - accuracy: 0.8066
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4324 - accuracy: 0.8066
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4532 - accuracy: 0.7959
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4370 - accuracy: 0.7871
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4301 - accuracy: 0.8027
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4319 - accuracy: 0.7910
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4268 - accuracy: 0.7979
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4096 - accuracy: 0.8154
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4738 - accuracy: 0.7734
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4411 - accuracy: 0.7842
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4410 - accuracy: 0.7754
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4437 - accuracy: 0.7881
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4273 - accuracy: 0.8047
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4030 - accuracy: 0.8193
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4604 - accuracy: 0.7725
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4432 - accuracy: 0.7744
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4299 - accuracy: 0.7969
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4276 - accuracy: 0.8037
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4138 - accuracy: 0.8076
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4009 - accuracy: 0.8105
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4273 - accuracy: 0.8086
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4049 - accuracy: 0.8193
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.3974 - accuracy: 0.8252
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4755 - accuracy: 0.7705
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4454 - accuracy: 0.7832
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4358 - accuracy: 0.7939
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4691 - accuracy: 0.7686
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4572 - accuracy: 0.7793
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4484 - accuracy: 0.7979
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4292 - accuracy: 0.8066
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4220 - accuracy: 0.8145
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4183 - accuracy: 0.8115
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4420 - accuracy: 0.7930
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4384 - accuracy: 0.7939
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4354 - accuracy: 0.7881
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4731 - accuracy: 0.7666
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4544 - accuracy: 0.7764
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4451 - accuracy: 0.7812
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4211 - accuracy: 0.8047
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4097 - accuracy: 0.8047
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4013 - accuracy: 0.8164
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4625 - accuracy: 0.7832
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4405 - accuracy: 0.7998
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4224 - accuracy: 0.8066
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4404 - accuracy: 0.7881
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4227 - accuracy: 0.7959
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4079 - accuracy: 0.8066
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4570 - accuracy: 0.7910
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4355 - accuracy: 0.8135
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4281 - accuracy: 0.7998
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4721 - accuracy: 0.7891
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4606 - accuracy: 0.7969
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4401 - accuracy: 0.8008
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4666 - accuracy: 0.7842
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4574 - accuracy: 0.7871
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4460 - accuracy: 0.7920
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4538 - accuracy: 0.7969
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4420 - accuracy: 0.8047
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4189 - accuracy: 0.8086
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4766 - accuracy: 0.7627
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4589 - accuracy: 0.7725
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4550 - accuracy: 0.7725
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4338 - accuracy: 0.8018
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4217 - accuracy: 0.8057
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4044 - accuracy: 0.8154
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4523 - accuracy: 0.7881
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4451 - accuracy: 0.7920
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4208 - accuracy: 0.8018
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4711 - accuracy: 0.7666
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4518 - accuracy: 0.7852
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4293 - accuracy: 0.7930
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4786 - accuracy: 0.7783
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4586 - accuracy: 0.7842
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4426 - accuracy: 0.7881
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4392 - accuracy: 0.8115
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4181 - accuracy: 0.8252
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4134 - accuracy: 0.8223
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4380 - accuracy: 0.7998
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4271 - accuracy: 0.8018
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4118 - accuracy: 0.8105
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4469 - accuracy: 0.7939
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4358 - accuracy: 0.7959
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4253 - accuracy: 0.8018
Epoch 1/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4574 - accuracy: 0.8018
Epoch 2/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4531 - accuracy: 0.8018
Epoch 3/3
32/32 [==============================] - 0s 3ms/step - loss: 0.4392 - accuracy: 0.7988
Epoch 1/3
19/19 [==============================] - 0s 3ms/step - loss: 0.4244 - accuracy: 0.8191
Epoch 2/3
19/19 [==============================] - 0s 3ms/step - loss: 0.4049 - accuracy: 0.8339
Epoch 3/3
19/19 [==============================] - 0s 3ms/step - loss: 0.3980 - accuracy: 0.8224

O modelo treinado incrementalmente pode ser salvo de maneira periódica (com base em casos de uso) e pode ser utilizado para inferir sobre os dados de teste nos modos online ou offline.

Referências: