The original project page on Google Code here: http://code.google.com/p/pyp/
git clone https://github.com/alexbyrnes/pyp.git
cd pyp
chmod u+x pyp
# Optionally: cp pyp /usr/local/bin
It's handy to put pyp into a directory on your path (for example /usr/local/bin) so you can type "pyp" instead of "./pyp".
####Background
Pyp, or The Pyed Piper, is an incredibly useful command line tool for:
- High volume transformations of unstructured data
- Operations that aren't available in Unix/Linux, or aren't easy
- Thinkers-in-python
####Usage
Filters
cat very_large_file.csv | pyp -L " len(p) > 5 " > only_long_lines.csv
Regular Expressions
cat very_large_file.csv | pyp -L " p.re('[0-9a-fA-F]*') " > only_hex_digits.csv
Compose multiple operations
cat very_large_file.csv | pyp -L " p.upper() | whitespace | p[:2] " > first_two_colums_uppercase.csv
Many more examples in the manual, and in examples.sh
####Running the Tests
python setup.py test
This will test pyp under multiple versions of python. If you only need to test a single version of python you can do this instead:
python setup.py test -a "-epy27"
####Making the C version (requires Cython)
make test
This will output a binary cyp and test it with a simple command.
#####What's New
- -L flag for large (> 50,000 line) files
- --DEBUG to debug output with line numbers and stack trace
- -D to output tab delimited text. The large file flag includes this automatically. Add -S to specify the delimiter.
- "cyp" compiled version
- Optimizations -- p.file, p.dir, and p.ext moved to p.file(), p.dir(), p.ext()