Changelog¶

[Unreleased]¶

BREAKING-CHANGES¶

API-Breaking¶

• the input definition for the tasks was changes to removed the naming restriction between dataset-column and hexagon-name
• the external REST-API endpoint to validate a token was removed and replaced by an internal function, which is used instead for all incoming api-requests
• changed password to passphrase, which is now required to be base64 encoded

Checkpoint-breaking¶

• added axons to be able to connect hexagons of the entire cluster with each other
• the synapse-block-links were moved into its own vector for cleaner separation and better sizes of the objects within the memory

Template-Breaking¶

• removed target-keyword from cluster-templates
• removed number of nodes from inputs and outputs from the cluster-templates

Database-Breaking¶

• the base64-representation of the passphrases is now used for the pw-hash inside the database, which makes all old hashes in the database invalid

Added¶

• added new workaround to be able to handle binary input-data
• added new output-modi to provide more output-options to combine multiple outputs bitwise into a bigger value
• added metrics for number of blocks and number of sections to cluster get output

Changed¶

• renamed repo from Hanami to OpenHanami
• tls-termination in kubernetes setup was moved from the ingress into a container within the same pod like the hanami-process
• number of input- and output-neurons now scales with the given data, so defining them hard inside of the cluster-template is not longer necessary
• moved converting user-context from the endpoints to a central position to avoid unnecessary redundant code
• moved cluster io-functions internally into the processing and backpropagarion files for the cpu-code
• item-buffer was internally changed to be more compatible for further attempts of GPU-support
• made internal processing more generic, to have the same worker-workflow for all coming backends
• use debug builds with ASan-check for tests of the CI (unit-, functional-, memory-leak-, sdk-api- and cli-api-tests)
• default-values for initial admin-credentials and token-key were removed from the helm-chart and are now marked as required
• synapse-tolerance was changed to static values

Fixed¶

• fixed memory-leak in DataSetFileHandle
• fixed memory violations found by the new added ASan-memory-check
• some cases of invalid http-requests had returned an 200 reponse with empty body, which was fixed
• changed token-key in helm-chart from configmap to secret
• fixed memory-corruption in cluster-resize, because a pointer was used after free
• fixed missing drawio-images in documentation when generating in CI

Removed¶

• the ansible-playbook was moved ot the archive and removed from the documentation and CI

v0.5.0¶

Release-Date: 2024-08-11

BREAKING-CHANGES¶

API-breaking¶

• tasks were updated to a more generic way
• request-results were removed from the databases and api, because results are now also stored as data-sets again
• renamed REST-API endpoints after removing the dashboard, because there is no separation between dashboard and commands anymore
• added version-numbers to the REST-API endpoints

Checkpoint-Breaking¶

• number of neurons per brick now grows dynamically and doesn't have to be defined in the cluster-templates anymore
• changed output-bricks to modifiy the size of the output-brick independently from the number of output-neurons
• updated buffer in clusters to remove the old byte-arrays
• new implementation for the checkpoints for the new data-structure of the clusters and for more and better error-handling
• header of checkpoint-files was changed from 512 byte to 4096 byte to have more space for additional configs

Dataset-Breaking¶

• new implementation for the datasets for a more generic handling of data and read-function to allow get specific sections from the data-set file
• header of dataset--files was changed from 512 byte to 4096 byte to have more space for additional configs

Template-Breaking¶

• cluster-templates were modified for the new dynamic neuron-allocation and the new output
• renamed bricks into hexagons

Database-Breaking¶

• database-tables now have a created_at-timestamp for the entry
• entries of the database-tables are now deleted anymore, but instead are only marked as deleted with a deleted_at-timestamp

Added¶

• added base64-type database-entries to easier store and get more complex strings from the database
• added documentation of the CLI-commands
• added function to download content of datasets
• added flag to the CLI to get every output as json-string
• added flag to the CLI to disable the TLS-check, when necessary
• database-tables now have proper functions to add and get structs from/to the database instead of only json-objects, which makes the code more explizit
• added new approach for time-series for input-data of csv-files
• added unit-tests for:
  • clusters
  • ckeckpoint create and restore
  • data-sets read and write
  • all database-tables
• improved CI-pipeline:
  • added commit-linter
  • added check for binary executables within the commited code
  • added secret-scan
  • build, use and upload helm-package
  • build, use and upload python-package of sdk
  • build, use and upload client
  • added CLI tests
  • automatic versioning of python- and helm-package
  • added compile-tests for different C++ compiler
  • added sdk-api-test with different Python versions
  • added integration-tests with different Kubernetes versions
  • added CodeQL-check as additional workflow
  • added OpenSSF-checks as additional workflow
• added dependabot-bot to repository

Changed¶

• datasets are now streamed internally and not fully loaded into the memory anymore, when creating a task, which is a massive reductions in the memory-consumption
• the registration of new endpoints within the code were updated by removing old leftovers
• the registration of new database-columns within the code were updated to look like the style of config-registrations and so on
• internal handling of tasks was updated for cleaner code
• endpoint to list projects of a user, were moved within the code
• renamed src-subdirectory Hanami to hanami to be more consistent to the rest
• split CUDA-kernel into multiple files
• replace the json submodule by the apt-package

Fixed¶

• fixed build of ARM64 docker-images, where were completely broken...
• fixed compiling of the C++ code under Ubuntu 24.04
• fixed csv-upload in python-sdk-lib
• fixed a value-limitation of 16bit, where it was not necessary
• fixed multiple commands within the CLI while adding automatic tests to the CI-pipeline
• fixed security-issues found by the new added CodeQL:
  • tls-check in CLI is not disabled be default anymore
  • fixed clear-text-logging of the password-input in the CLI
  • fixed a range-check in a for-loop within the backend

Removed¶

• the common-directory in Hanami was removed and most of the functions were moved to the hanami_common-lib
• disabled CUDA-kernel, because at the moment it doesn't brings a performance increasement, but only slows down the development (will be added in the future again, after finding a better use-case for the GPU)
• disabled dashboard and moved it to the archive subdirectory (will be added in the future again, after a complete refactoring of the code)
• removed old OpenCL-kernel

v0.4.1¶

Release-Date: 2024-03-25

Added¶

• provide docker-images also for arm64 architecture
• new steps were added to the CI-pipeline:
  • added automatic build and push of amd64 and arm64 docker-images to docker hub
  • added automatic build and push for the documentation to docker hub
  • added SDK-API-Test
  • added integration-tests for kubernetes- and ansible-setup with amd64 docker-images
  • added flake8 to lint python-code
  • added ansible-lint to lint ansible-code

Changed¶

• reduce cpu usage in idle state by blocking inactive threads
• use earthly as new build-script and replace old pre-build docker-images in CI
• use random instead of static salt-value for password-hashes
• websocket-connections in SDK use now async-functions

Fixed¶

• fixed linting-errors in python- and ansbile-scripts
• fixes to run code in virtual machines, where reading system information had broke the setup

v0.4.0¶

Release-Date: 2024-01-21

BREAKING-CHANGES¶

Checkpoint-Breaking¶

• Complete restructure of the core to improve the performance of the CUDA-kernel by enabling more capabilities for parallel processing.
• Synapse-blocks are not limited to a single cluster-scoped buffer, but instead are stored in a global buffer, which is shared by multiple cluster.
• The number of used synapse-blocks is now independed from the number of neuron-blocks of a brick. They scale now better over time. This way the number of neurons in the clsuter-template are only a maximum value. This way it is easier to too high values in the cluster-template are no pain-point anymore.

API-Breaking¶

• removed c++ version of the SDK

Added¶

• multi-threading, so now multiple cpu-threads can process the same cluster at the same time
• new endpoint to list logical hosts (cpu's and gpu's)
• new endpoint to move a cluster between cpu and gpu
• re-added the spiking neural network-feature for future testing-purpose
• re-added a connection-distance of more the one for future testing-purpose
• python version of the SDK
• basic cli-client in go
• new entry in config to define a location for temporary files while uploading
• new cleanup-loop for tempfiles, which deleted incative temporary files after a certain amount of time, to remove file of a broken upload-process
• the reduction-process was re-added to limit the amount of used memory while learning.

Changed¶

• physical host is now separated into logical hosts (a cpu and a gpu are handled as 2 different logical hosts)
• replaced old api-test-tool by a new python-script, which use and test the new python version of the SDK
• moved from qmake to cmake as build-tool for the c++ code

Removed¶

• removed GET-requests from audit-log
• removed c++ version of the SDK

v0.3.3¶

Release-Date: 2023-11-17

Fixed¶

• fixed broken login to dashboard
• fixed broken internal error-log and the error-log API-endpoint

v0.3.2¶

Release-Date: 2023-11-10

Added¶

• basic Ansible-playbook was added in order to deploy the project without kubernetes
• in kubernetes-setup:
  • now use cert-manager to generate certificates
  • ingress was added for ssl-termination of https-connections
  • persistent volume was added to persist data like database within the kuberntes-setup
  • node label was added to define the node, where hanami should be deployed by kubernetes
• clang-format file was added with a git pre-commit hook
• clang-format check and basic cppcheck were added to the ci-pipeline
• first contribution-guide and code-styling guide was added
• example-configs for testing purpose and guide to setup local test-environment
• at start there is now a check if the directories for the checkpoints and dataset, defined by the config, even exist

Fixed¶

• fixed compile-error when trying to build on ARM64 architecture
• fixed compile-error in cude-code with the standard nvidia-toolkit in ubuntu 22.04
• fixed broken json-strings in javascript-sdk
• fixed false error response-codes in API

Changed¶

• moved old readme files into the normal documentation
• use clang++ instead of g++ as compiler
• non-critical API-errors, like for example a 404 (not found) when searching for an unknown id, doesn't produce an internal error-output anymore

Removed¶

• ssl-termination was removed from backend-API, because it will be done by another service in front of the API, like nginx

v0.3.1¶

Release-Date: 2023-09-24

Changed¶

• replaced json-parser by third-party library https://github.com/nlohmann/json
• updated internal registration of config-values
• updated internal registration of cli-arguments

Fixed¶

• fixed bug in synapse-segmentation, which had impact in samller tests

Removed¶

• Removed unused and deprecated checkpoint create and finalize endpoints

v0.3.0¶

Release-Date: 2023-09-05

BREAKING-CHANGES¶

All breaking¶

• Move the entire project from a micro-service architecture to a more monolithic architecture with only one executable but still multiple libraries, to reduce unnecessary complexity and increase performance and reliability of the program. Also it makes the developing process easier and faster, with is necessary in regard of my limited time-resources.
• Removed entire segment-layer between the clusters and bricks from core structure. This layer was originally intended for the separation of multiple threads, but now there is another strategy planed

API-Breaking¶

• REST-API endpoint for the generation of the documentation was removed
• Renamed snapshot to checkpoint and learn to train

Database-Breaking¶

• fixed sizes of some database-columns for user- and project-id's

Checkpoint-Breaking¶

• Bigger rework of the core structs to get rid of the update-positions struct

Added¶

• documentation of config-entries and database-schemas can now also be generated as marddown-documents
• error-codes are now also written into the REST-API documentation
• new config-entry to enable the extremely experimental CUDA-processing
• more then one thread is now created for the core-processing, to be able to process multiple clusters in parallel
• the CUDA-version now also works with Checkpoints (create and read)

Changed¶

• Rename repository/project from Hanami-AI to Hanami
• Renamed libraries to hanami_... and namespaces of the libraries to Hanami, because the originally naming and structure was from the time, when they were separate repositores and desired also for other projects. The new naming makes the names shorter and easier.
• replace custom jwt lib by jwt-cpp (https://github.com/Thalhammer/jwt-cpp)
• REST-API documentation is now generated as OpenAPI-specification and added to the documentation via swagger-ui
• merged networking-libraries in context of the renaming
• config-handler was internally restructured for the new generator of the config-documentation
• registration of fields in API-endpoints was updated for cleaner code
• Button in dashboard for the geneartion of the Documentation was removed and is now done by a new CLI flag.
• merged entire code into one single docker-image
• better segmentation of the synapses

Fixed¶

• solved all compiler-warnings (disabled libraries not included)
• fixed stupid memory leaks in API and task-handling
• fixed handling in database-requests to separate correctly between an internal-error and a not-found to give the correct HTTP response to the user
• after a restart of the backend, all clusters are not removed from database at the start to avoid broken clusters, because cluster are in-memory and don't survive a restart
• positioning of header-texts in dashboard was fixed

Removed¶

• Removed OpenCL-kernel for the moment. The CUDA-variant is enough for testing and easier to update.

v0.2.0¶

Release-Date: 2023-03-15

Breaking-Changes¶

API-Breaking¶

• use now custom syntax for cluster- and segment-templates instead of json
• use protobuf-messages to send payload of files
• merge endpoints to create train- and request-tasks for csv- and mnist-files into one endpoint
• creating a task doesn't require the explicit selecting of the data-set type. This is now read from the data-set metadata

Checkpoint-Breaking¶

• changed internal data-structure of the core in order to make GPU-support possible without race-conditions

Changed¶

• merged all repositories (source-code, documentation, build, deploy) into this main-repository here
• made token-key and registry in helm-chart configurable
• made token expire-time configurable in Misaki
• core-internal renamings of structs and functions
• use dropdown-menu instead of single buttons in table-entries within the dashboard
• use nested namespaces in sakura- and hanami-layer
• changed a huge amount of map-iterators to c++17 style

Added¶

• Added experimental GPU-support with CUDA and OpenCL (per default hard in source-code disbabled at the moment)
• internal tokens requestable form Misaki
• API-endpoints and database-table to request logs with page-selector
• print audit-logs in dashboard

Fixed¶

• fix random breaks of the websocket under high load in the Torii
• fixed memory-corruption in segment-header of the clusters in Kyouko
• solved uninitialized warning in item-buffer
• fixed stupid memory-leak when sending internal messages

Removed¶

• Removed additional namespaces of the libraries in the common-layer
• Disabled reduction-process of the neural-networkds for the moment.

v0.1.0¶

Release-Date: 2022-10-18

Added¶

• First experimental prototype version with very basic implementation of following features:
  • first incomplete implementation of the core-concept of a dynamic network with:
    • creates connections while learning
    • has nearly no upper limit for inputs
  • capable to train MNIST-Datasets and basic CSV-files
  • creating neuronal networks based on templates in json-format
  • authentication with username + password and JWT-token
  • creating multiple user and multiple projects in one system
  • basic role-system
  • automatic creation of REST-API-documentation in PDF-, RST- or Markdown-format
  • creating checkpoints of neuronal networks and restore them again
  • thread-binding for all components for more optimal usage of the CPU
  • PoC to change CPU-frequency based on load
  • measuring of power consumption, thermal production and frequency of the CPU
  • Dashboard with HTML, Javasript and CSS with rendering on client-side
  • sdk-library in C++ and Javascript (incomplete)
  • Build-scripts to build docker-images for all components
  • basic Helm-chart to deploy all components on a kubernetes

Repositories¶