* replace dagger html entity

* strip extra spaces at beginning of xml document

* log progress while matching figure image to each pdf page

* yield empty image object match to keep total

* added simple MultiLevelCache

* implemented disk cache

* added TestRegisterPickleFunction

* increased default memory cache size

* moved get_image_array_with_max_resolution

* added --pdf-scale-to argument

* convert pdf from local pdf

* show progress when loading pdf images

* workaround for auto-scaling

* log progress while checking for existing files

* renamed --skip-errors to --ignore-unmatched-graphics

* implemented --skip-errors

* run test_should_annotate_using_jats_xml using beam

* fixed MapOrLog usage when skipping errors

* moved bbox main to separate module

* renamed test module to match main module under test

* conditionally write dummy copy of xml file

* renamed to output_json_path

* added coords attribute

* using namespace for coords attribute

* add namespace to nsmap

* process pdf and xml file lists

* allow sub directories in output path

* make output_annotated_images_path relative to output

* make sure that output directory is created

* use write_bytes in favour of explicit makedirs (cloud ready)

* make --pdf-base-path required

* added pipeline

* not extending ABC due to serialization errors

* log pipeline options

* added test to check serialization

* changed super import to avoid one of the serialization errors in Dataflow

* moved most functionality to separate module

* added libgl1 to setup.py

* added PreventFusion

* minor import grouping

* added TransformAndCount

* reverted super __init__ call

* use parse args, not ignoring unknown args

* expose all of the worker arguments

* added BoundingBoxScoreSummary

* added TestGetBoundingBoxMatchScoreSummary

* implemented algorithm to adjust final bounding box

* specify cache key for object keypoints

* made image id required for lower level functions

* added logging to find bounding boxes

* 2nd it to find bounding box without bounding box

* use fixed size when calculating image similarity

* calculate structural similarity using skimage

* fixed crop_image_to_bounding_box

* output score in JSON

* optionally output images with bounding boxes

* display bbox label inside if not enough space above

* sort by score, then key points

* fixed cache issue by using explicit cache key prefix

(otherwise ids may have been reused after memory being freed)

* raise GraphicImageNotFoundError

* allow skipping errors

* enable debug logging for tests

* added cli scaffolding

* extract images from pdf

* fixed type hint

* added bounding box to_list

* converted bounding box to named tuple

* added tests for validate

* implemented bounding box intersection

* implemented finding bounding boxes of single image

* added test for smaller partial image

* added libgl1 for open cv

* linting: use with statement for Popen

* added support for multiple image files

* added support for xml files

* join graphic href with xml dirname

* renamed cv2 to cv

* using ObjectDetectorMatcher

* moved funtions to image object matching module

* added TestGetObjectMatch

* added ImageObjectMarchResult

* added test_should_match_smaller_image

* added test_should_match_smaller_rotated_90_image

* fixed typo ImageObjectMatchResult

* moved object_detector_matcher parameter down

* added get_image_list_object_match

* added support for gzipped files

* indent output json

* added annotation file_name

* using sample image as fixture

* reduce size of sample image

* added save_images_as_pdf

* added test using multiple images

* prefer better match based on keyword match count

* added --debug cli arg

* handle case where no homography can be found

* use category based on parent element

* added formula type

* added related_element_id

* added logging tqdm

* added info

* resize larger image to make finding bounding boxes faster

* convert to gray scale to make finding boxes even faster

* made max width height configurable

* cache image features

* increased default max width height

* added tests for get_image_array_with_max_resolution

* allow max width height to be zero

* make grayscale conversion optional

* disabled max resolution by default

* added mypy dependency

* added dev-mypy

* added mypy make target

* declare EMPTY class prop

* removed incorrect tensors type hint

* added type hint to excluded_tokens

* removed unused ProcessedWrapper

* replacing backports.tempfile with builtin

* added T_ArgumentParserOrGroup

* fixed iter_tokenized_tokens return type hint

* added types-requests

* added type to DEFAULT_ANNOTATORS

* removed blank line

* ignore distutils import

* replaced T_Element with etree.ElementBase

* changed type check back to etree._Element

* replaced project_tests.sh

* removed second mypy make target dependency

* Bump pylint from 2.8.3 to 2.10.2

Bumps [pylint](https://github.com/PyCQA/pylint) from 2.8.3 to 2.10.2.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/main/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/v2.8.3...v2.10.2

)

---
updated-dependencies:
- dependency-name: pylint
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* make installing dependencies more predictable

* downgrade numpy due to conflict with apache beam

* downgraded numpy further due to conflict with tensorflow

* linting: use dict literal

* linting: pass in encoding to open function

* linting: pcoll renamed to input_or_inputs

* linting: iterate over list

* linting: use from .. import

* added pyarrow as explicit dependency

* downgrade google-cloud-bigquery

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Daniel Ecer <de-code@users.noreply.github.com>

* initial create vocabulary utility

* extract vocabulary from embeddings

* renamed to --output-word-count-file

* added main call

* extracted iter_tokenized_tokens

* avoid empty tokens

* using tokenizer from delft

* optionally sort by count

* added file list support

* added support for remote files

* added limit argument

* added fsspec dependency

* optionally use multi threading or processing

* included full github link

* renamed to create_vocabulary

* moved to tools vocabulary

* filter embeddings

* renamed to embeddings

* using fsspec to open embeddings file when extracting

* use fsspec when filtering embeddings

* document tools

* added link to tools.md

* removed usage of six

* removed six dependency

* Bump pylint from 2.3.1 to 2.4.4

Bumps [pylint](https://github.com/PyCQA/pylint) from 2.3.1 to 2.4.4.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/master/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/pylint-2.3.1...pylint-2.4.4

)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* removed explicit astroid dependency

* linting

* more flake8 linting

* explicit pylint dependency

Co-authored-by: Daniel Ecer <de-code@users.noreply.github.com>

* ugraded to python 3

* upgrade pylint and pytest

* replaced StandardError

* exclude useless-object-inheritance

* python3 compatibilities uncovered by linting

* fixed tests

* fixed more python3 test incompatibilities

* added pylint check

* added pylintrc to docker image

* reduced accessive apache beam debug logging

* configured pylint, addressed linting

* enabled flake8 checks

* downgrade pycodestyle to 2.3.1 due to error

* switch to 4 spaces indent

* autopep8

* more flake8

* added new line to .flake8

* added sciencebeam-utils and sciencebeam-alignment dependency

* replaced local alignment module with sciencebeam-alignment

* replaced local beam_utils with sciencebeam-utils

* replaced local utils with sciencebeam-utils for modules that have moved

* removed tools that have been moved to sciencebeam-utils

* removed utility functions that have moved to sciencebeam-utils

* updated readme

added option to calculate sample class weights (instead of using same class weights for all samples)