How to calculate bits per character of a string? (bpc) to read

 

http://stackoverflow.com/questions/17797922/how-to-calculate-bits-per-character-of-a-string-bpc

up vote1down votefavorite

 

A paper I was reading, http://www.cs.toronto.edu/~ilya/pubs/2011/LANG-RNN.pdf, uses bits per character as a test metric for estimating the quality of generative computer models of text but doesn't reference how it was calculated. Googling around, I can't really find anything about it.

Does anyone know how to calculate it? Python preferably, but pseudo-code or anything works. Thanks!

python algorithm machine-learning nlp entropy

share|improve this question

edited Jul 23 '13 at 1:41 jogojapan 24k42859

asked Jul 22 '13 at 21:40 Newmu 48619

Are you talking about the stuff defined in C CHAR_BITtigcc.ticalc.org/doc/limits.html#CHAR_BIT ? –  woozyking Jul 23 '13 at 0:31

Nope, this is related to information theory and entropy, not actual bit size. – Newmu Jul 23 '13 at 0:46

add comment

2 Answers

activeoldestvotes

up vote5down voteaccepted

Bits per character is a measure of the performance of compression methods. It's applied by compressing a string and then measuring how many bits the compressed representation takes in total, divided by how many symbols (i.e. characters) there were in the original string. The fewer bits per character the compressed version takes, the more effective the compression method is.

In other words, the authors use their generative language model, among other things, for compression and make an assumption that a higheffectiveness of the resulting compression method indicates highaccuracy of the underlying generative model.

In section 1 they state:

The goal of the paper is to demonstrate the power of large RNNs trained with the new Hessian-Free optimizer by applying them to the task of predicting the next character in a stream of text. This is an important problem because a better character-level language model could improve compression of text files (Rissanen & Langdon, 1979) [...]

The Rissanen & Langdon (1979) article is the original description ofarithmetic coding, a well-known method for text compression.

Arithmetic coding operates on the basis of a generative language model, such as the one the authors have built. Given a (possibly empty) sequence of characters, the model predicts what character may come next. Humans can do that, too, for example given the input sequence hello w, we can guess probabilities for the next character: o has high probability (because hello world is a plausible continuation), but characters like h as in hello where can I find.. or i as in hello winston also have non-zero probability. So we can establish aprobability distribution of characters for this particular input, and that's exactly what the authors' generative model does as well.

This fits naturally with arithmetic coding: Given an input sequence that has already been encoded, the bit sequence for the next character is determined by the probability distribution of possible characters: Characters with high probability get a short bit sequence, characters with low probability get a longer sequence. Then the next character is read from the input and encoded using the bit sequence that was determined from the probability distribution. If the language model is good, the character will have been predicted with high probability, so the bit sequence will be short. Then the compression continues with the next character, again using the input so far to establish a probability distribution of characters, determining bit sequences, and then reading the actual next character and encoding it accordingly.

Note that the generative model is used in every step to establish a new probability distribution. So this is an instance of adaptive arithmetic coding.

After all input has been read and encoded, the total length (in bits) of the result is measured and divided by the number of characters in the original, uncompressed input. If the model is good, it will have predicted the characters with high accuracy, so the bit sequence used for each character will have been short on average, hence the total bits per character will be low.

---

Regarding ready-to-use implementations

I am not aware of an implementation of arithmetic coding that allows for easy integration of your own generative language model. Most implementations build their own adaptive model on-the-fly, i.e. they adjust character frequency tables as they read input.

One option for you may be to start with arcode. I looked at the code, and it seems as though it may be possible to integrate your own model, although it's not very easy. The self._ranges member represents the language model; basically as an array of cumulative character frequencies, so self._ranges[ord('d')] is the total relative frequency of all characters that are less than d (i.e. a, b, c if we assume lower-case alphabetic characters only). You would have to modify that array after every input character and map the character probabilities you get from the generative model to character frequency ranges.

share|improve this answer

edited Jul 23 '13 at 11:01

answered Jul 23 '13 at 1:40 jogojapan 24k42859

2

Excellent introduction and explanation, thank you! –  Newmu Jul 23 '13 at 4:00

The model I'm working on generates a probability distribution for the next character given a fixed length of previous characters, so it looks like it'll be some work but I should be able to figure it out. –  Newmu Jul 23 '13 at 4:14

@Newmu It should definitely be possible. But I agree it will be easy to make mistakes. You'll need to test this carefully after implementing it. (Unfortunately I have no insider knowledge about arcode. The code seemed to be relatively easy to understand, which is why I suggested this module. There are others, too, though. If you do a search for "python arithmetic coding" or similar, you may be able to find implementations more suitable.) –  jogojapan Jul 23 '13 at 4:31

add comment

up vote0down vote

The sys library has a getsizeof() function, this may be helpful?http://docs.python.org/dev/library/sys

share|improve this answer

answered Jul 23 '13 at 0:57 ChrisProsser 2,5691620

add comment

Not the answer you're looking for? Browse other questions tagged python algorithm machine-learning nlp entropy or ask your own question.

asked	7 months ago
viewed	199 times
active	7 months ago

Community Bulletin

event

Moderator candidates' answers to your questions– ends in 6 hours

event

2014 Community Moderator Election– ends in 6 hours

blog

Podcast #55 – Don’t Call It A Comeback

• Systems Developer - Hong Kong
  Jane StreetHong Kong
• Shanghai - Python Engineer for Big Data Start-up
  www.caigen100.comShanghai, China
• Software Developer (Functional Programming) - Hong Kong
  Jane StreetHong Kong
• More jobs near Shenzhen...

Related

3

How Do You Categorize Based On Text Content?

228

How does the Google “Did you mean?” Algorithm work?

2

How to calculate tag-wise precision and recall for POS tagger?

21

How do I approximate “Did you mean?” without using Google?

3

How to recognise a particular user in a long multi-user internet chat log?

1

How do I approach this named-entity classification task?

99

How does Apple find dates, times and addresses in emails?

3

Python NLTK: How to retrieve percentage confidence in classifier prediction

1

How to implement Bag of words feature hashing in python?

0

Calculate entropy of a file

Hot Network Questions

•  

  Is it recommended or advised against to play sounds for warning and error messages?

•  

  CV for leaving academia after PhD

•  

  How would you describe the grammatical case that allows this joke?

•  

  How does a commercial flight pilot handle turbulence?

•  

  QuickSort of Comparable[]

•  

  How to make or add options for packages or functions

•  

  Extends JFrame vs. creating it inside the the program

•  

  Why is printing "B" dramatically slower than printing "#"?

•  

  Should I list skills on my résumé if I have no interest in using them again?

•  

  Is cooling really exponential?

•  

  Customised Java UTF-16

•  

  Playing a loyal character without it being boring

•  

  Why is the moment of inertia for a hollow sphere higher than a uniform sphere?

•  

  Table with Tabu & siunitx not working

•  

  How does different spacing affect the unary operator?

•  

  Why do political polls have such large sample sizes?

•  

  Why are parents so concerned about not letting the babies sleep with them?

more hot questions

question feed

about help badges blog chat data legal privacy policy jobs advertising infomobile contact us feedback

TECHNOLOGY			LIFE / ARTS	CULTURE / RECREATION	SCIENCE	OTHER
Stack Overflow Server Fault Super User Web Applications Ask Ubuntu Webmasters Game Development TeX - LaTeX	Programmers Unix & Linux Ask Different (Apple) WordPress Answers Geographic Information Systems Electrical Engineering Android Enthusiasts Information Security	Database Administrators Drupal Answers SharePoint User Experience Mathematica more (14)	Photography Science Fiction & Fantasy Seasoned Advice (cooking) Home Improvement more (13)	English Language & Usage Skeptics Mi Yodeya (Judaism) Travel Christianity Arqade (gaming) Bicycles Role-playing Games more (21)	Mathematics Cross Validated (stats) Theoretical Computer Science Physics MathOverflow more (7)	Stack Apps Meta Stack Overflow Area 51 Stack Overflow Careers

site design / logo © 2014 stack exchange inc; user contributions licensed under cc by-sa 3.0 withattribution required

rev 2014.2.25.1396