undate development meeting 2024-06-04

[rlskoeser]

Agenda

goals:

• decide on EDTF scope and compliance
• decide if and how to implement database fields for undate objects
• decide schedule for upcoming monthly meetings

discussion items:

• confirm EDTF scope/testing:
  • dates only, not time?
  • how do we confirm compliance with the portions of the EDTF spec we want to support?
• django/database approaches; review similar partial date / EDTF dates implementations & rule out / decide on approach
  • python-edtf django field: stores EDTF as string and parses into text label and upper/lower fuzzy/strict
  • Shakespeare and Company Project partial date - stores a date and date precision flag; custom descriptor to combine the two into a partial date object (similar to undate); custom django filter knownyear; allows working with the dates more in the database (e.g. django admin date_hierarchy) but you have to be aware that they are partial dates and not regular dates
  • Princeton Geniza Project partial/converted dates - stores a calendar, string of date in original format, string of date in standardized format; dates are stored as ranges, e.g. YYYY/YYYY or YYYY-MM-DD/YYYY-MM-DD; we do calculations in python and sorting/filtering in Solr, not currently leveraging the database for any date logic
• questions:
  • how valuable/important to support date functionality in the database? (sorting, filtering, querying)
  • what range of years do we want to support (broader question); python datetime max year is 9999, postgres date max year is 5874897
  • can we target postgres or is it important to be db agnostic?

[rlskoeser]

attending: @rlskoeser @ColeDCrawford

notes

Cole has been working a lot on v5 python-edtf release, should be out soon.

• updated all the tests, CI/CD, overhauled grammar, improved parsing (pyparsing; packrat?), + tests
  • no way to get a human readable date from a string; hard problem (EDTF to human readable; it does support the reverse but it's decent, not great); struggles with parsing ranges, gets first part but not always the second part; struggles with qualifiers, doesn't know which one to apply or which part of the date
• uses date struct rather than python datetime, because of the year problem
  • harder for development; provides abstraction wrapper
  • the year range isn't a problem for most humanities data / humanities projects
• implications for undate? overlap? things we could we do better / differently?

edtf implementation - agreed on: dates only, but no time

db implementation discussion

python-edtf:

• was handling human readable and edtf date parsing; separate now
• parses upper/lower bounds (both strict and fuzzy); stores all the pieces (?)
  • Cole thinks it does create floats for filtering in the db
  • look at v5 branch; includes a small test django app, useful to check
• previous versions only stored the string and re-parsed all the time (slow)

mep-django / S&co:

• datetime & flag to identify which parts of the date are reliable
• human readable date template that's undate/partial date aware
• admin filters that take advantage of the date information
• need to handle more cases / feature flags (known, partially known, uncertain, etc)
• maybe json field could work for the flags?
• custom template tag to display human readable date in specified format but that is aware of which parts are unknown
  • this would be super useful; better if not specific to django, e.g. some equivalent of datetime.strptime / strftime that handles different formats

geniza:

• stores calendar, original date (character), standardized date (character: ISO date or date range)
  • calendar stuff is very useful, EDTF ignores that entirely (different problem)

postgres vs broader db support, e.g. sqlite?

• how important is broad db support? easier adoption?
• we use postgres but others might want something lighter-weight
• sqlite json1 extension - so maybe json is fine!

these implementations are a good start, know the pain points and have some directions

• sketch out a proof of concept, see how it works with these projects, iterate, review
• multiple fields underneath is fine as long as you can initialize as a single field and it's handled for you

answers to questions in agenda:

• how valuable/important to support date functionality in the database? (sorting, filtering, querying)
  • very useful, should definitely do
• what range of years do we want to support (broader question); python datetime max year is 9999, postgres date max year is 5874897
  • if it's easier to stick with datetime; covers 99% of cases; only handles a certain level of EDTF, but less important; 0 and level 1 covers the majority of cases
• postgres vs broader db support, e.g. sqlite?
  • how important is broad db support? easier adoption?
  • we use postgres but others might want something lighter-weight
  • sqlite json1 extension - maybe json field is fine!

testing/validating edtf compliance, performance?

• maybe can borrow/adapt from python-edtf tests
  • tests parsing date to bounds: https://github.com/ixc/python-edtf/blob/v5/edtf/parser/tests.py
  • natural language parsing (not as strong): https://github.com/ixc/python-edtf/blob/v5/edtf/natlang/tests.py
  • performance testing?
    • benchmarks: https://ixc.github.io/python-edtf/dev/bench/
    • PR that added them: 50 add performance tests ixc/python-edtf#57
• also important to test the django fields; examples here: https://github.com/ixc/python-edtf/blob/v5/edtf_django_tests/edtf_integration/tests.py - minimal django app for testing purposes