* Added most x86 cryptographic instructions
* Fixed popcnt test
* Fixed asm_ops_x64_2.c3t test
---------
Co-authored-by: Christoffer Lerno <christoffer@aegik.com>
* stdlib: implement `std::compression::zip` and `std::compression::deflate`
- C3 implementation of DEFLATE (RFC 1951) and ZIP archive handling.
- Support for reading and writing archives using STORE and DEFLATE
methods.
- Decompression supports both fixed and dynamic Huffman blocks.
- Compression using greedy LZ77 matching.
- Zero dependencies on libc.
- Stream-based entry reading and writing.
- Full unit test coverage.
NOTE: This is an initial implementation. Future improvements could be:
- Optimization of the LZ77 matching (lazy matching).
- Support for dynamic Huffman blocks in compression.
- ZIP64 support for large files/archives.
- Support for encryption and additional compression methods.
* optimizations+refactoring
deflate:
- replace linear search with hash-based match finding.
- implement support for dynamic Huffman blocks using the Package-Merge
algorithm.
- add streaming decompression.
- add buffered StreamBitReader.
zip:
- add ZIP64 support.
- add CP437 and UTF-8 filename encoding detection.
- add DOS date/time conversion and timestamp preservation.
- add ZipEntryReader for streaming entry reads.
- implement ZipArchive.extract and ZipArchive.recover helpers.
other:
- Add `set_modified_time` to std::io;
- Add benchmarks and a few more unit tests.
* zip: add archive comment support
add tests
* forgot to rename the benchmark :(
* detect utf8 names on weird zips
fix method not passed to open_writer
* another edge case where directory doesn't end with /
* testing utilities
- detect encrypted zip
- `ZipArchive.open_writer` default to DEFLATE
* fix zip64 creation, add tests
* fix ZIP header endianness for big-endian compatibility
Update ZipLFH, ZipCDH, ZipEOCD, Zip64EOCD, and Zip64Locator structs to
use little-endian bitstruct types from std::core::bitorder
* fix ZipEntryReader position tracking and seek logic ZIP_METHOD_STORE
added a test to track this
* add package-merge algorithm attribution
Thanks @konimarti
* standalone deflate_benchmark.c3 against `miniz`
* fix integer overflows, leaks and improve safety
* a few safety for 32-bit systems and tests
* deflate compress optimization
* improve match finding, hash updates, and buffer usage
* use ulong for zip offsets
* style changes (#18)
* style changes
* update tests
* style changes in `deflate.c3`
* fix typo
* Allocator first. Some changes to deflate to use `copy_to`
* Fix missing conversion on 32 bits.
* Fix deflate stream. Formatting. Prefer switch over if-elseif
* - Stream functions now use long/ulong rather than isz/usz for seek/available.
- `instream.seek` is replaced by `set_cursor` and `cursor`.
- `instream.available`, `cursor` etc are long/ulong rather than isz/usz to be correct on 32-bit.
* Update to constdef
* Fix test
---------
Co-authored-by: Book-reader <thevoid@outlook.co.nz>
Co-authored-by: Christoffer Lerno <christoffer@aegik.com>
* math: implement discrete and continuous distributions
Implement a comprehensive set of continuous and discrete probability
distributions with support for PDF, CDF, inverse CDF, random sampling,
mean, and variance calculations.
The following distributions are implemented:
* Normal
* Uniform
* Exponential
* Chi-Squared
* F-Distribution
* Student t
* Binomial
* Poisson
* update releasenotes.md
* Formatting
---------
Co-authored-by: Christoffer Lerno <christoffer@aegik.com>
* Optimized adler32 implementations.
- Adapted adler32 implementation from Crypto++ public domain library.
- Added unit tests for adler32 hashing algorithm.
* tabified adler32 implementation to match stdlib.
* Formatting to be consistent. Make unrolling use macro.
---------
Co-authored-by: soerlemans <sebasoerlemans+git@gmail.com>
Co-authored-by: Christoffer Lerno <christoffer@aegik.com>
* split `test_hash_vector_macro` into per-type functions
it went from a single 18367 instructions function, to eleven separate
functions with (1000 to 1600) instructions each
thats 15649 instructions total, 2718 less instructions overall.
```
$ build/c3c compile-test --suppress-run -O1 --print-large-functions -D
SLOW_TESTS test/unit/ 2>&1 | sort -n | grep "test_hash_vector"
1041 instructions found in
std::core::builtins:test_hash_vector_int128
(/home/mb/scripts/c3c/test/unit/stdlib/core/builtintests.c3) - function
is very long.
1041 instructions found in
std::core::builtins:test_hash_vector_uint128
(/home/mb/scripts/c3c/test/unit/stdlib/core/builtintests.c3) - function
is very long.
1305 instructions found in std::core::builtins:test_hash_vector_long
(/home/mb/scripts/c3c/test/unit/stdlib/core/builtintests.c3) - function
is very long.
1305 instructions found in
std::core::builtins:test_hash_vector_ulong
(/home/mb/scripts/c3c/test/unit/stdlib/core/builtintests.c3) - function
is very long.
1453 instructions found in std::core::builtins:test_hash_vector_int
(/home/mb/scripts/c3c/test/unit/stdlib/core/builtintests.c3) - function
is very long.
1453 instructions found in std::core::builtins:test_hash_vector_uint
(/home/mb/scripts/c3c/test/unit/stdlib/core/builtintests.c3) - function
is very long.
1565 instructions found in
std::core::builtins:test_hash_vector_short
(/home/mb/scripts/c3c/test/unit/stdlib/core/builtintests.c3) - function
is very long.
1565 instructions found in
std::core::builtins:test_hash_vector_ushort
(/home/mb/scripts/c3c/test/unit/stdlib/core/builtintests.c3) - function
is very long.
1629 instructions found in std::core::builtins:test_hash_vector_char
(/home/mb/scripts/c3c/test/unit/stdlib/core/builtintests.c3) - function
is very long.
1629 instructions found in
std::core::builtins:test_hash_vector_ichar
(/home/mb/scripts/c3c/test/unit/stdlib/core/builtintests.c3) - function
is very long.
1663 instructions found in std::core::builtins:test_hash_vector_bool
(/home/mb/scripts/c3c/test/unit/stdlib/core/builtintests.c3) - function
is very long.
```
* made the `test_hash_vector_internal` generic
* optimize `test_ct_intlog2` while still covering all 128 bit positions
* refactor whirlpool to reduce code bloat
replaced the fully unrolled round loop with a runtime loop, reducing
instruction count by 80k in `process_block` and yielding aprox 30%
performance boost due to improved cache locality.
* use compile-time arrays for `test_ct_intlog2`
* codepage: add single-byte code page support
Add std::encoding::codepage with a shared engine for converting between
single-byte code pages and UTF-8 using table-driven mappings.
Introduce generated tables and wrappers for several code pages[1] each
exposing encode/decode helpers built on a common CodePageTable
structure.
The mapping data is generated by cpgen[2] from the Unicode Consortium’s
published code page mapping files and follows the Unicode standard’s
interpretation of control characters (abstract characters) rather than
historical VGA glyph shapes.
[1] Code page overview/groups:
DOS/OEM code pages (legacy PC):
cp437 cp737 cp775 cp850 cp852 cp855 cp857 cp860 cp861 cp862 cp863
cp864 cp865 cp866 cp869 cp874
Windows code pages (ANSI/Windows):
cp1250 cp1251 cp1252 cp1253 cp1254 cp1255 cp1256 cp1257 cp1258
ISO/IEC 8859 series (Latin/Regional):
iso_8859_1 iso_8859_2 iso_8859_3 iso_8859_4 iso_8859_5 iso_8859_6
iso_8859_7 iso_8859_8 iso_8859_9 iso_8859_10 iso_8859_11 iso_8859_13
iso_8859_14 iso_8859_15 iso_8859_16
[2] github.com/konimarti/cpgen
Signed-off-by: Koni Marti <koni.marti@gmail.com>
* codepage: change encoding format, streamline api
* Use enum to collect the data.
---------
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Co-authored-by: Christoffer Lerno <christoffer@aegik.com>
* [stdlib] Add PEM Encoding/Decoding Module
* release notes
* Removed some unnecessary macro usages. Fixed memory handling with headers.
* Make end of line a parameter. Internal encode method -> function. Use more tmem. Remove t-functions.
* Update API
---------
Co-authored-by: Christoffer Lerno <christoffer@aegik.com>
Co-authored-by: Christoffer Lerno <christoffer.lerno@gmail.com>
* Merge AsciiCharset CT/non-CT Functions
* release notes
* incorporate helpful review feedback
* re-separate 'create_set' and 'contains' but keep 'combine_sets'; update tests
* tabs (annoying IDE)
* Restored old code verbatim for smaller diff. Split combine_sets into easier to macro/function for runtime / macro version, this also allows for more easy type checks.
---------
Co-authored-by: Christoffer Lerno <christoffer@aegik.com>
* Fixed macro hash_vec(vec): it was checking only 8 bytes
* use $sizeof() directly as it is correct now
* Added unit test for vector types
Tests that hashes match for same values and that they don't match
when changing a single bit on any of the vector elements
* Changed hardcoded value for the compiler option
* Release note
---------
Co-authored-by: Christoffer Lerno <christoffer@aegik.com>
* netbsd fail on error
- Group each BSD test in its own subshell. Without this, if an error
occurs, it exits, but we don't know "where" it failed
- added *usesh: true* to BSD for faster copying into the virtualbox VM
- Some test_suite_runners wheren't correctly printing. added
--no-terminal
- For testing I've changed *build-msvc* from Debug to RelWithDebInfo
Signed-off-by: Manuel Barrio Linares <mbarriolinares@gmail.com>
* separate runs
Signed-off-by: Manuel Barrio Linares <mbarriolinares@gmail.com>
* fix typo
Signed-off-by: Manuel Barrio Linares <mbarriolinares@gmail.com>
* Split msvc-debug into separate workflow
Run msvc-debug only on maintainer pushes, keep CI fast and still catches
errors.
Signed-off-by: Manuel Barrio Linares <mbarriolinares@gmail.com>
* Fix build and test errors on NetBSD
* Set proper sizes for various types used for threads
* Add num_cpu function
* Add sysctl constants
* Allow testproject to build on NetBSD
* Tweaks to the linker code so that it adds correct flags and finds dynamic linker on NetBSD
* bsd: force system linker test
Signed-off-by: Manuel Barrio Linares <mbarriolinares@gmail.com>
* system linker on openbsd
remove lld from netbsd
Signed-off-by: Manuel Barrio Linares <mbarriolinares@gmail.com>
* use cc linker in openbsd, and re-add lld for netbsd
Signed-off-by: Manuel Barrio Linares <mbarriolinares@gmail.com>
* openbsd: fixes
- implement num_cpu() for OPENBSD
- testproject: add openbsd
- linker.c: fix for openbsd
- bsd refactor main.yml and more fixes
* openbsd: fix for unit tests and test_suite_runner
openbsd now passes all tests except the 'dynlib-test' which yields
'relocation error' when running `cc test.c -L. -ladd -Wl,-rpath=.`
Signed-off-by: Manuel Barrio Linares <mbarriolinares@gmail.com>
* openbsd: guard/disable unit test test_ct_intlog2()
* build-msys2-clang: add all tests that build-msvc has
Signed-off-by: Manuel Barrio Linares <mbarriolinares@gmail.com>
* Fix Windows import library generation by adding /IMPLIB support and properly declaring static_lib_name()
Signed-off-by: Manuel Barrio Linares <mbarriolinares@gmail.com>
* build-msys2-mingw: add all tests that build-msvc has and enable it
configure CMake to link LLVM and LLD statically
* Revert "Revert windows"
This reverts commit 197f82d829.
* win-llvm 21.1.8
uses the RelWithDebInfo+Assertions
* refactor(CI): Centralize test logic and fix Docker execution
This commit refactors the GitHub Actions workflow to improve
maintainability and correct a flaw in the Docker build.
- Ensures Docker based tests run *inside* the generated container, not
on the host. This also fixes environment isolation and user permission
errors.
- Centralizes all common test steps (examples, libs, unit tests, etc)
into a single, auto-detecting Bash script: `scripts/tools/ci_tests.sh`.
- Centralize all release artifact packaging into a single
`scripts/tools/package_build.sh`
- fixes using correct paths for test_suite_runner when running on
windows under bash.
- This significantly reduces YAML duplication, making `main.yml` more
readable and easier to maintain.
This one supersedes pull/2677, and includes the fixes for:
- fix openbsd and netbsd
- added win-llvm 21.1.8 and fixed CMakeLists.txt
- added full unit-tests and suite for build-msys2-mingw and
build-msys2-clang
* macos: add/fix build for llvm (19,20,21)
Signed-off-by: Manuel Barrio Linares <mbarriolinares@gmail.com>
* ci: run tests in temp dir to avoid workspace pollution
Executes ci_tests.sh in a disposable temporary directory by copying
resources/ and test/ folders, preventing build artifacts from cluttering
the source tree.
This makes it easy to run `scripts/tools/ci_tests.sh` locally on any
platform.
Signed-off-by: Manuel Barrio Linares <mbarriolinares@gmail.com>
* add ilammy/msvc-dev-cmd@ and Ninja
we can simplify the windows runner and run everything in bash
Signed-off-by: Manuel Barrio Linares <mbarriolinares@gmail.com>
* windows: drain subprocess stderr before join to avoid test_suite_runner hang
* move last test to `ci_tests.sh`
thus completing the centralization of tests in `ci_tests.sh`
* Fix Windows CI cache
Updates the cache key to include hashes of CMakeLists.txt and the
workflow file.
Signed-off-by: Manuel Barrio Linares <mbarriolinares@gmail.com>
* `package_build.sh` runs in temp dir to avoid workspace pollution
- add cleanup
* nix: use `ci_tests.sh`
* remove annoying build-mac warnings
This now checks if package already installed trying to prevent
annoying warnings in the github ci:
Fixes:
```
build-mac (Release, 19)curl 8.17.0 is already installed and up-to-date.
To reinstall 8.17.0, run: brew reinstall curl
```
* refactor test_suite_runner
- less allocations, cleaner code, more maintainable.
- now can be killed and the temp dirs are correctly cleaned up, and the
output is correct.
* main.yml: bsd: change to using rsync, is twice as fast.
* default to `--wincrt=dynamic` when `--sanitize=address` on windows
Default to dynamic, as static ASan is removed in LLVM 21+ for Windows
https://github.com/llvm/llvm-project/pull/107899
Signed-off-by: Manuel Barrio Linares <mbarriolinares@gmail.com>
* Formatting (indent with tab, align with space). Fix K&R braces. Style: keep all cases multiline if one is.
---------
Signed-off-by: Manuel Barrio Linares <mbarriolinares@gmail.com>
Co-authored-by: Christoffer Lerno <christoffer@aegik.com>
Co-authored-by: Alex Garrett <limit.ordinal17@gmail.com>
* Add unit tests for HMAC 256 based on RFC 4231
* Formatting, shorten initializers.
---------
Co-authored-by: Christoffer Lerno <christoffer@aegik.com>
* blake3: initial unit test passing!
* typo
* add key derivation macros; vanilla unit tests; working to test 18 atm
* mark it here - all tests passing o_o
* finish first-round unit tests - will add more if necessary
* add crypto shootout bench entertainment
* tests: add XOF unit w/ seek; assert NO finalization
* add another to finalizations unit test
* add all BLAKE3 scaffolding for later SIMD optimizations
* irksome
* tabs
* tabs2
* extra documentation / contracts
* extra detail
* try to make things a bit more arch-neutral
* release notes
* Formatting
---------
Co-authored-by: Christoffer Lerno <christoffer@aegik.com>
* Add NetBSD Support
Includes:
- Hints to find non-compatibility libc functions
- Struct and constant definitions for sockets, polling, etc.
- Changes to the linker code to work around some quirks in the NetBSD dynamic linker
- A target triple for netbsd aarch64 so llvm builds/links the compiler properly on this platform
* Updated releasenotes and some compacting
---------
Co-authored-by: Christoffer Lerno <christoffer@aegik.com>
* ChaCha20 implementation, first pass
* fix bug with clone_slice when length is 0
* final ChaCha20 crypto tidying
* final adjustments; add benchmark
* add guards everywhere else or w/e
* stdlib 'i++' conformity
* release notes & security warning updates
* update tests; cleanup; default counter should be 0 not 1
* remove prints in test file
* add extra unit tests for unaligned buffers
Co-authored-by: Manu Linares <mbarriolinares@gmail.com>
* one final alignment test
* nice contraction of tests w/ some paranoia sprinkled in
* nearly double the efficiency of chacha20's transform
Co-authored-by: Manu Linares <mbarriolinares@gmail.com>
* fix memory leak in test case
* improve one of the unit tests to cover more cases
* greatly simplify chacha20 'transform'
Co-authored-by: Manu Linares <mbarriolinares@gmail.com>
---------
Co-authored-by: Manu Linares <mbarriolinares@gmail.com>
