MemoPen
- A media connecting man, paper and computer -
Table of Contents
- Toward rich media
- MemoPen
- System of MemoPen.
- Reconstructiong software
- Evaluation - Is it really used? -
- Conclusion and the future of MemoPen
- Column: Making of MemoPen
- Acknowledgement
- References
History of human begins is the history of media. We can think the media
itself creates the history because only the matter recorded in media can be
dealt with as the history, except fossils and kitchen middens having
remained naturally. The followings are the historically remained inventions
on recording techniques. The great inventions, for examples, are as
follows.
- Papyrus
- Paper
- Letter press
- Photograph (still)
- Record (sound)
- Picture (movies)
It goes without saying that how much these inventions have made our lives
rich. But we would like you to notice that these inventions direct to
dynamic from static that is addition to time axis. Of course (1)~(4) do not
deal with data containing time axis. However, with the invention of paper,
indispensable plane for reading was offered and as uniform letters could be
arranged precisely by printing techniques, a great amount of letter
information can be put into brains through vision. We can say this
indicates to realize some kind of letter broadcasting like an electric
bulletin board. The flowing, of course, is not letters but vision.
By the link set up in all directions some kinds of flow is introduced to
the documents written in HTML (Hyper Text Markup Language) used for
popular WWW (World Wide Web). The keys of popularity of WWW are the
individual can browse freely the flow and he can easily makes the flow and
send it for the world. HTML documents can be displayed enough by the
popular personal computer, but it is interesting the capacity of
communication line, which is the flowing amount of information per an hour,
is not satisfied. Really favorable environment for computer user is not to
use the huge and powerful computer but to use the high speed network
freely. The age to realize this seems to be coming soon.
Anyhow the flow from the poor media (static) to the rich media (dynamic)
seems to be necessary and not to be stopped. Here we use the word "rich"
in two meanings that are having time axis and the relationship between the
matter represented by the media is dynamic. But in the present situation,
even the writing process, the most fundamental way to record, can not be
recorded naturally. One of our members has been thinking it a very
significant problem for these ten years. This consciousness of question,
that is the question whether the missing clink, equal to the part
surrounded by the frame of Fig. 1 exists or not,
leads MemoPen project.
This paper is the report challenging for five years to the requirement,
which we want to record the writing work, the basic of man's intellectual
work, naturally with time axis.
Figure 1. History of media
Typical way to use the MemoPen, a media recording naturally the process
of writing, is shown in Fig. 2. One of the key
points is we can use it quite same as the ordinary pen. For example, we
do not need the digitizing sheet like PDA, nor special paper written
down. We can write on not only paper, but also even the palm of hand and
table cloth or so. We also can use it as an ordinary pen. This means we
do not need to learn how to use it and this MemoPen is the media
standing on the extended line of work to write on the paper with pen,
which man has been used to for a long time. It also makes the best use
of all the characteristics of paper; cheap, free size, handy to carry
(able to bend or ball up).
However, there are some problems on paper as a media which is too bulky to
preserve and difficult to retrieve, to reuse and to revise the content
written down. It is, of course, the computer to cover these faults. If the
letter is made into character code by the technique of recognition of
handwriting, the problems of preservation, reuse and revise will be solved
at once. Now the research on the retrieval of digital data of handwriting
itself is being made, it will be possible to retrieve to the illustration
written by hand with its outline in future.
Here we suppose some of you may doubt whether this dreaming media can be
realized or not, and so let us show the basic idea. The idea is quite
simple. Super miniature CCD camera set on the tip of a pen is only to take
in and record the images near the tip of a pen (called partial image)
every per a unit time.
After finishing, removable memory pack (serves battery pack)
is taken away and writing data is sent to the computer. By reconstructing
the whole images from the partial images with the computer, the record of
writing work with time axis is to be over. This idea is so simple that you
may have some questions by contraries as follows.
- Is it really possible to reconstruct the whole image from
the partial image in the small scope around the tip of a pen?
- Is it able to be realized with the present or near future
implementation technique?
- When it is achieved, how long can it record?
- Unbelievable unless you will not see the MemoPen itself.
From now answering these questions, we explain the system of MemoPen
in detail.
Figure 2. Way to use MemoPen
System of MemoPen is constructed with MemoPen and reconstructing software.
We explain as follows; hardware, software and latest prototype in order.
3.1. Hardware
The concept of MemoPen is shown in Fig. 3 and block
diagram in Fig. 4. MemoPen is constructed with pen
axis, lens, CCD, pressure sensor, microprocessor, memory and battery.
When we use MemoPen to write letters, the image around the tip of pen
focused on CCD through lens, that is the image of handwriting, can be
taken in with the computer every certain sampling time and memorized as
the handwriting data in the memory contained inside the pen. And with
the pressure sensor the writing pressure is detected and memorized in
the memory in the same way. According to the pressure sensor signal, we
can distinguished whether the MemoPen is writing or moving in the air.
For CCD is set on the tip of MemoPen and has a small scope, the
handwriting data taken in with MemoPen will not be the whole image but
partial image. The example of the partial image is shown in
Fig. 5.
Figure 3. Scheme of MemoPen
Figure 4. Block Diagram of MemoPen
Figure 5. Partial Iamges (example)
3.2. Software
By connecting MemoPen with the computer through
the interface unit, the handwriting data memorized in MemoPen and output
of pressure sensor are taken into the computer. The handwriting data
memorized in MemoPen is the partial image data written down and can be
reconstructed with the reconstructing software. When we use MemoPen, as
pen itself taking the partial image in moves, the absolute coordinate
does not exist and absolute position of handwriting written down can not
be recognized.
However, the sampling time is so short that the piling common parts may
exist among the part of images continuously with time. From these common
parts the relative location among the partial images is sought and the
whole images are reconstructed by piling up the part of images with no
contradiction.
3.3. Latest prototype of MemoPen
The latest prototype of MemoPen is shown in Fig. 6.
The whole length of MemoPen is 280mm. The weight is about 70g. It is
five minutes for MemoPen to record the letters. The scope of MemoPen is
about 20mm by 15mm. As the part images taken in is 32 by 40 pixels, the
resolution is 1.6 pixel/mm high and 2.7 pixel/mm long. The prototype can
take in the partial images of binary images by the sampling of ten
snapshots per second.
Figure 6. Latest Prototype of MemoPen
We make the model of reconstructing the whole images from the partial
images with the reconstructing software as the four processes of piling
shown in Fig. 7. Cell means the partial images
recorded with MemoPen and it is the most basic image. Act means
the partial reconstructing images gathered depending on the common
piling handwriting among continuing cells. That is, cells are put on
the same act and gathered as long as the relationship with just before
the cell is distinguished. When the common handwriting between act and
cell disappears, new act is constructed. Having constructed new act
means the relationship with the just before cell could not be
distinguished by changing the line or so. As the letters like "
" (Hiragana, Japanese letter pronounced as ku)
are began to write from the right comparatively, sometimes it happens
that the relationship with just before the cell in case of writing
horizontally will not be distinguished. This is typically the cause to
change the act.
Figure 7. Cell, Act, Scene and Page
Between the acts, which are once made because the relationship could not
be distinguished, the common handwriting will appear as the writing
proceeds. For example, when we move from the last letter of the first
line to the first letter of the second line by changing the line, new
act is to be made. As we proceed to write the second line, the part
under the first line will appear on the upper part of the second line.
We can gather the acts with the common handwriting between the acts.
This is the scene. All the acts having the piling each other are put on
as a scene. This is shown in Fig. 8.
Figure 8. Scene Construction (example)
As these exists no common handwriting between the scenes, we can not
put on one more scene. But, as there are the images constructed with
the handwriting of writing work using MemoPen, they are gathered as
the page. Changing the page is to be done by the user's clear
direction. The directions, for examples, are considered to switch on
to begin the writing and to switch off finishing the writing, or to
direct with some special gesture. The act and scene were named after
the play.
Reading this paper to this stage, you may have a doubt in spite of
understanding the idea whether it is really used or not. Instrument
which seems powerful if it could exist, is first recognized in the
world by proving its practicability, making a prototype. If we omit
this process, we are only the mad scientist. In the followings, we
introduce the part result of experiment made in order not to finish as
mad scientists. You can refer to the reference 3 in detail.
- Experiment 1
- The success rate of reconstructing for the size of letters
written with MemoPen.
The size of letters: 8mm, 10mm, 15mm
- Experiment 2
- The success rate for the space between two letters.
Space between letters: 0mm, 3mm, 5mm
- Experiment 3
- The success rate for the speed of writing letters.
Necessary time of writing for one letter:
1.2sec, 1.5sec, 2.0sec, 2.6sec, 3.3sec
- Experiment 4.
- The success rate for practice times of MemoPen before the
experiments.
a) The writers having different practice times write
the same sample.
b)The writers having different practice times write freely.
The kinds of letters written down were Hiragana, Katakana,
Kanji, alphabet, and numerical expressions. A few of illustrations
were written. The number of letters of each sample is about 3 to 12
letters by 2 to 5 lines and written both horizontally and lengthwise.
From experiment 1 and 2, as for
the latest prototype, the letter about 10 mm was proved to be the most
suitable for reconstructing from the point of success rate and
resolution. And the scope of CCD proved to need the size more than
(letter + space between letters) x 2.
From experiment 3, we made it sure that the
writing speed does not much influence on
the reconstructing rate. This shows the sampling rate to take in the
cell of the latest prototype is sufficient.
From experiment 4, the writers who practice with MemoPen
about 10 to 20 times, speaking to the number of letters, about 200 to
300 letters can get relatively the high success rate. In the present
situation the skill like this standard is necessary. The result of
experiment 9 is indicated in Fig. 9 and the
reconstructed scene rightly in Fig. 10.
Figure 9. Result of Experiment 4(a)
Figure 10. Whole Image rightly reconstructed
We explained about MemoPen, a new media to supplement the problem of
control with computer, as making best use of the merits of paper concerned
intellectual work. By using MemoPen it becomes possible to record the
handwriting of letters, memo and illustrations together with time axis.
For example, when we consider the recognition of letters, the
handwriting as ink data added time axis is needless to say more useful
than the still images taken into with scanner. As an example, consider
to distinguish "
" (Katakana pronounced as
"so") and "
" (Katakana pronounced as "n"),
an d the handwriting corrected with the doubled lines or crossing
lines. This is the characteristic only the media with time axis can
have.
By making the writing process clear, the thought process, which is the flow
of more abstract activity, is to be proved later. For example, even if you
are not writers, you will be interested in the creative process of great
writers and musical composers.
Lately, we conclude this paper by showing the examples of junction
seemed to be separated in Fig. 1. One of the
understandable examples is the record of sound by using MemoPen. It is
difficult at the present stage of technique to retrieve to the recorded
conference with tape recorder. And so its effective use is difficult.
For the reason, we can indicate the point: The most important part of
the conference is only the small part, even if it is so significant
conference, and the most part is not connected with the theme when we
listen to the conference later. (But there exists some indication the
most part like that is important) Therefore to listen through the whole
is painful. We can not separate the important part from others' on the
conference.
However, thinking further more, man records the important parts as memos
during the conference. At this time it is possible, if we use MemoPen, to
warp to the time around from the handwriting, for the time stamp is added
to the handwriting. Of course the same thing can be said in the case of
video. This is nothing but adding tags by using MemoPen to sound or movies
of continuous media.
The technique that changes the time or time axis data which are
difficult to be caught or dealt with for the busy man in the present
age to the data seen by man and put on the link freely is one example
of MemoPen in future.
1) Before the first prototype (~1991)
We experiment to reconstruct by making a simulator forming the imaginary
images with the motion of mouse because we do not have the prototype.
This experiment was done at random because it was not clear for MemoPen to
have how wide scope and how fine resolution. So the students taking charge
of the experiment seem to be unable to realize what the MemoPen is.
Thinking now, it was a pity, but it could not be helped because this
project itself was groping then.
2) The first prototype (on November, 1992)
The first prototype having the construction which sets up lens beside the
tip of a pen and leads its focus point to the Panasonic color T.V. camera (
330,000 elements) with the use of optical fiber is to be completed. Using
this, we begin to experiment the reconstructing software.
The most serious problem is the number of elements of CCD. We want to take
in about 1000 pixels finally like the number of pixels of the third
prototype is 40 pixel long x 32 pixel wide. But the high performance handy
camera is so popular recently that CCD with a few number of pixel is not
available. For this reason, after recording in the VHS video, we take in
the computer and thin out.
Taking in the images was also the cause of difficulty. Now if we equip some
optional board and hard disc drive by G byte to the workstation, we can do
most of the things and in our laboratory we can take in the real time
images easily. But at this time our laboratory does not have the experience
in images and we ask another laboratory which has the equipment to take in
the real time images to record in CD possible to write in. This data is
1.2M byte by one piece of partial image, and this is a great deal at that
time.
Also it was not clear how well reconstruction we could do only by the
images of CCD and we did the experiment to use Hall effect device to obtain
its movement, fixing the magnetic ball on the tip of ball-point pen. For
the ball does not move while the pen move in the air, the position is to be
decided by the CCD images during transferring in the air. As for continuous
stroke, we confirmed to reconstruct almost perfect only by Hall effect
device. To polish the ball as the perfect sphere and magnetize it was more
difficult than we expected.
3) The second prototype (on May, 1993)
In order to avoid the getting worse of images by optical fiber, the second
prototype (See Fig. 11) mounted the solid TV camera beside the ball-point
pen was completed.
At this stage, an opinion appeared; we may be able to get the movement
of pen precisely if there is an identification on the paper. Though
there is an opposite opinion not to think it a proper way, we were to
experiment using graph paper anyhow. At the time of developing
algorithm to remove the distortion of lens and drawing out mesh of
graph paper printed in pale blue with color filtering, the third
prototype was completed and the part image was made binary image of 40
x 32 pixel. It is quite ironical. For it is difficult to separate
pre-printed mesh at the level of this resolution, we stop the research
of using graph paper. At the result it became to reconstruct
handwritings without this algorithm, and so we have no problem....
In order to extend the recording time and making optical system
small, we decided to make the third prototype having about 3cm scope
diameter and 40x32 pixel of the partial image. For this reason the
software group changeover the way for the research of reconstructing
algorithm using partial images of low resolution.
Figure 12. Partial Image on Graph Paper
4) The third prototype (on October, 1993)
The third prototype is almost the final form except the problems of size
and connected with interface box by the tail (electric wire). Interface
system has the detachable IC memory card, and when the writing is over, the
data is transferred by putting in the memory card reader of PC. We use CCD
(about 4000 elements) for interphone manufactured by Sanyo by thinning it
out at the level of one fourth. The appearance is seemed like a drill as
the dentist shaves the teeth and it is a little horrible. But its color is
smart gold.
The completion of the third prototype moved our software group very much.
By that time we could not realize the partial image. But now the whole
image of system is made real and the research of software began to proceed.
It is a bad point for software researchers not to do their best until the
real thing is appeared. At the end of that year the first version of
reconstructing software on X-Window system, which corresponds with the
third prototype, and soon the idea of hierarchical construction of cell,
act, scene and page.
5) The fourth trial machine (on April, 1994)
We have been examining the use of several kinds of sensors(Hall effect
device, motion sensor) and at last we can confirm to get enough
reconstructing precision with the combination of CCD and pressure sensor.
In the fourth prototype we improved optical system of the third prototype.
But it cost so expensive that we might be able to buy a high class car.
Owing to this, we could get good improvement. Then it may be cheap. We
showed this in SIGHI of Japan Information Processing Society in July of
that year, and in the video session of WISS'94 by Japanese Society of
Software Technology in December of that year.
6) The fifth prototype (on April, 1995)
The fifth prototype has the detachable memory pack (serves battery pack)
and its form is same as an ordinary pen. The size is a little thicker than
the high class fountain pen of Monblanc. The software was transported to
MS-Windows and made its debut in CHIC'95 hold in Denver, U.S.A. There we
heard " Great idea! " but we were sorry that we couldn't distinguish by our
English ability whether this saying was real praise or lip service. We also
heard the proposal to buy at once by $100.
We thank to Toshio Taguchi, the president of Yashima Electric Company as
the original planner. The hardware is the fruit of joint research with the
project team of MemoPen in Yashima Electric Company. The software is the
fruit of Human Interface & Media Group of Agusa Laboratory in Nagoya
University including Shinji Nabeshima. We thank to all the members. We also
thank to Prof. Michihiko Mino, Kyoto University and Prof. Shigeo Sugimoto,
University of Library and Information System, who made their efforts to
take in images at the first stage. Without their assists, our project would
not have even started.
- K.Weber, A.Poon and T.Cass: Scriber: A Tool for Searching Digital
Ink,Proc. of CHI '95, pp.252-253, 1995
-
S.Nabeshima,
H.Nakamura,
S.Yamamoto and
K.Agusa:
The way to reconstruct the whole image from the
partial image system with memopen (in Japanese).
SIG94-HI-55-1, IPSJ, pp.1-8 July.
-
S.Nabeshima:
Research on the way to reconstruct the whole image
from the partial image system with memopen (In Japanese).
Master's thesis, Nagoya University, 1995
-
S.Nabeshima,
S.Yamamoto,
K.Agusa and T.Taguchi:
MEMO-PEN: A New Input Device. Proc. of CHI '95, pp.256-257, 1995
Agusa Lab. Home Page
Dept. of Information Engineering Home Page