�
�c1 �
�~ � d dl Z d dlmZ d dlmZ d dlmZ d dlmZm Z m
Z
mZmZ ddl
mZ ddlmZmZmZmZ dd lmZ dd
lmZ ddlmZmZmZmZmZ ded
e
e fd�Zded
e
e fd�Z e� � ded
e
e fd�� � Z! e� � ded
e
e fd�� � Z" ee�� � ded
ee#e#f fd�� � Z$ d&de
e de#d
e
e fd�Z%dede
e d
e&fd�Z'ded
e
e fd�Z(de
e d
efd�Z)ded
efd �Z* ed!�� � d'ded#e&d$ee d
efd%�� � Z+dS )(� N)�IncrementalDecoder)�Counter)� lru_cache)r �Dict�List�Optional�Tuple� )�FREQUENCIES)�KO_NAMES�LANGUAGE_SUPPORTED_COUNT�TOO_SMALL_SEQUENCE�ZH_NAMES)� is_suspiciously_successive_range)�CoherenceMatches)�is_accentuated�is_latin�is_multi_byte_encoding�is_unicode_range_secondary�
unicode_range� iana_name�returnc �� ��� t | � � rt d� � �t j d� | � � � � j } |d�� � }i �d�t
dd� � D ]h}|� t |g� � � � }|rAt |� � }|��9t |� � d u r|�vrd�|< |