pcaptoparquet

This is a package for converting pcap files primarily to parquet format. CSV and JSON formats are also supported.

Installation

To install the package, run the following command:

pip install pcaptoparquet

Or with a virtual environment:

python -m venv <your_venv>
source <your_venv>/bin/activate   # In windows: .\<your_venv>\Scripts\activate
python -m pip install --upgrade pip
python -m pip install pcaptoparquet

Usage

Command Line Interface

To see all available options:

pcaptoparquet -h

Note 1: Portable executable can be created with pyinstaller but this executable has not been fully tested. Check the Makefile for more information.

Note 2: CLI interface was not fully tested in Windows environment. Unit testing is only instrumented for Linux.

Programming Interface

The pcaptoparquet package provides the E2EPcap class for converting pcap files to different formats. Here's how you can use it:

1. Import the E2EPcap class from the pcaptoparquet package:

from pcaptoparquet import E2EPcap

2. Create an instance of E2EPcap with the path to your pcap file:

pcap = E2EPcap('path_to_your_pcap_file')

Replace 'path_to_your_pcap_file' with the actual path to your pcap file.

3. Use the export method of the E2EPcap instance to convert the pcap data to a different format:

pcap.export(format='parquet', output='output_directory')

The format parameter specifies the output format. In this example, we're converting the pcap data to parquet format.

The output parameter specifies the directory where the output file will be saved. Replace 'output_directory' with the actual path to your output directory.

This is a basic example of how to use the pcaptoparquet package. Depending on your needs, you might need to use additional methods or parameters.

Refer to pcaptoparquet_cli.py for a more complex example of use. In particular, refer to extensibility options such as application protocol implementations and post-processing callbacks associated to E2EConfig class. Full examples included in tests folder (config and callbacks subfolders).

Contributing

Contributions are welcome. Please follow these steps to contribute:

1. Fork the repository.
2. Create a new branch (git checkout -b feature-branch).
3. Make your changes.
4. Run the tests and code quality checks to ensure everything works correctly.
5. Commit your changes (git commit -am 'Add new feature').
6. Push to the branch (git push origin feature-branch).
7. Create a new Pull Request.

Please make sure to update tests as appropriate.

Development Installation

After cloning the repository, you can set up your development environment:

python -m venv <your_venv>
source <your_venv>/bin/activate   # In windows: .\<your_venv>\Scripts\activate
python -m pip install --upgrade pip
python -m pip install -e .
python -m pip install black isort pyright flake8 Flake8-pyproject mypy tox coverage build twine

or

make venv

Testing

The pcaptoparquet package includes a suite of tests to ensure its functionality. These tests are located in the tests directory.

To run the tests, you'll need tox, which is a tool for automating testing in multiple Python environments.

Here's how you can run the tests:

tox

or

make test

This command will run all the tests in the tests directory and display the results in the terminal.

If you make changes to the pcaptoparquet code, please make sure to run the tests and ensure they all pass before submitting a pull request.

Coverage is also available but only informational for now:

make coverage

Code Quality Checks

The pcaptoparquet project uses several tools to ensure code quality:

• black: for code formatting
• isort: for sorting imports
• pyright: for type checking
• flake8: for linting
• mypy: for static type checking

You can run these checks using the following commands:

black --check pcaptoparquet_cli.py pcaptoparquet tests
isort --check-only pcaptoparquet_cli.py pcaptoparquet tests
pyright pcaptoparquet_cli.py pcaptoparquet tests
flake8 pcaptoparquet_cli.py pcaptoparquet tests
mypy pcaptoparquet_cli.py pcaptoparquet tests

or

make check

To automatically fix formatting issues:

make fix

License

This project is licensed under the BSD-3-Clause License. See the LICENSE file for more details. Copyright 2025 Nokia.

DeepWiki

Additional documentation can be found at pcaptoparquet.