Handwriting Recognition for Math: Write on Your Screen

1. The Challenge of Mathematical Handwriting

Recognizing handwritten text is a problem that computer science has been working on for decades. For ordinary language -- letters, words, and sentences -- the technology has reached impressive levels of accuracy. But mathematical handwriting presents an entirely different set of challenges that push recognition technology to its limits. Mathematics is not just a linear sequence of characters; it is a two-dimensional language where spatial relationships carry meaning.

Consider a simple fraction: the numerator sits above a horizontal line, with the denominator below. A superscript turns a number into an exponent, fundamentally changing the meaning of the expression. The square root symbol wraps around its contents. Summation notation stacks limits above and below the sigma. Matrices arrange numbers in grids. These spatial arrangements are not decorative -- they are the grammar of mathematics, and any recognition system that fails to capture them accurately will produce nonsensical results.

Beyond spatial complexity, mathematical handwriting involves an enormous vocabulary of symbols. In addition to the standard Latin and Greek alphabets, math uses hundreds of specialized symbols: integral signs, partial derivatives, set membership operators, logical connectives, arrows of various types, and many more. Each of these must be distinguished from visually similar alternatives. The difference between an italic lowercase L, the number one, and a vertical bar can be razor-thin in handwriting, yet each has a completely different mathematical meaning.

There is also the challenge of individual handwriting variation. Every person writes differently, and mathematical handwriting tends to be particularly idiosyncratic. Students often develop shorthand notations, skip symbols they consider obvious, or write in ways that make perfect sense to them but would confuse anyone else. A robust recognition system must handle this variation gracefully, learning to interpret each user's personal writing style while maintaining mathematical accuracy.

For students, the stakes are high. If a recognition system misinterprets what they have written, it could mark a correct answer as wrong, provide irrelevant hints, or generate confusion that undermines the learning experience. Conversely, if the system can accurately read handwritten math, it opens up a remarkably natural way to interact with digital learning tools -- writing on a screen just as one would write on paper, but with the added benefit of instant AI-powered feedback and guidance.

At Kedmathic, we recognized early that solving the mathematical handwriting recognition problem was essential to creating the learning experience we envisioned. Students should not have to translate their mathematical thinking into a keyboard-based interface. They should be able to think in math, write in math, and have the technology meet them where they are. This conviction has driven our deep investment in handwriting recognition technology and the development of our Smart Canvas feature.

2. How OCR Works for Math Expressions

Optical Character Recognition, or OCR, is the technology that converts images of text into machine-readable data. For mathematical expressions, OCR must accomplish two interrelated tasks: recognizing individual symbols and understanding the spatial structure that gives those symbols meaning. Modern math OCR systems use deep learning -- specifically, convolutional neural networks and transformer architectures -- to accomplish both tasks simultaneously.

The process begins with the raw input: either a sequence of pen strokes captured in real time (online recognition) or a static image of handwritten math (offline recognition). In online recognition, the system has access to the temporal order of strokes -- which line was drawn first, the direction and speed of each stroke -- which provides valuable additional information. Offline recognition, used when scanning a photo of written work, must extract all information from the image alone, making it a harder problem but one that is essential for features like textbook scanning.

For online recognition, the system processes each stroke as it is drawn, maintaining a running hypothesis about what the student is writing. This is more complex than it might sound. When a student draws a horizontal line, the system cannot immediately determine whether it is a minus sign, a fraction bar, or the top of a plus sign whose vertical component has not yet been drawn. The system must maintain multiple possible interpretations simultaneously and resolve the ambiguity as more strokes are added, using both the visual appearance of the marks and the mathematical context to determine the most likely interpretation.

The structural analysis phase is where math OCR diverges most dramatically from ordinary text recognition. While text flows in one direction (left-to-right in English, right-to-left in Hebrew), mathematical expressions spread across two dimensions. The system must build a tree-like structure that captures the hierarchical relationships between symbols: this group of symbols is the numerator of that fraction, this symbol is the exponent of that variable, these elements are inside that square root. Getting this structure right is just as important as recognizing the individual symbols correctly.

Training these neural networks requires enormous datasets of handwritten mathematical expressions, annotated with their correct interpretations. Building these datasets is itself a significant challenge, as each expression must be labeled not just with its symbolic content but with its full structural representation. At Kedmathic, we have augmented publicly available datasets with our own collection of student handwriting, ensuring that our models perform well on the kinds of expressions that Israeli students actually write -- which may differ from the academic datasets commonly used in research.

Error correction is a critical final step. Even the best recognition models make occasional mistakes, so the system includes multiple layers of validation. Mathematical grammar rules can catch impossible combinations of symbols. Context from the exercise being worked on can resolve ambiguities -- if the student is solving a quadratic equation, a symbol is more likely to be an "x" than a multiplication sign. And real-time feedback allows the student to verify the recognition result and correct any errors before proceeding, ensuring that the system's interpretation matches their intent.

3. PencilKit and MathPix Integration

Kedmathic's handwriting recognition system is built on two powerful technological foundations: Apple's PencilKit framework and the MathPix recognition engine. Together, these technologies enable a handwriting experience that is both natural to use and highly accurate in its mathematical interpretation. Understanding how they work together reveals the engineering effort behind what appears to the student as an effortless experience.

PencilKit is Apple's native framework for capturing and rendering digital ink. It provides the low-level infrastructure for turning Apple Pencil movements into beautiful, responsive strokes on the screen. PencilKit handles the subtle physics of digital writing: pressure sensitivity that makes strokes thicker when you press harder, tilt detection that affects the stroke width when you angle the pencil, and palm rejection that prevents accidental marks when your hand rests on the screen. These details are crucial for creating a writing experience that feels natural rather than clinical.

On top of PencilKit's stroke capture, Kedmathic layers its own intelligence. We track not just the final appearance of each stroke, but its dynamics: the speed at which it was drawn, pauses mid-stroke, and the temporal relationship between strokes. This temporal data is enormously valuable for recognition. For instance, a student who draws a circle in one smooth motion is more likely writing a zero or the letter O, while one who draws a circle and then adds a vertical line through it might be writing the Greek letter phi or the empty set symbol. These distinctions are invisible in a static image but clearly distinguishable in stroke data.

MathPix provides the core mathematical recognition engine. Originally developed as a research project and now a leading commercial math OCR service, MathPix specializes in converting images and handwritten input into structured mathematical notation. The engine can recognize an extraordinarily wide range of mathematical content, from basic arithmetic through advanced calculus, linear algebra, and beyond. It outputs results in LaTeX format, a standard mathematical typesetting language that Kedmathic can then parse, display, and process.

The integration between PencilKit and MathPix is where significant engineering effort has been invested. The challenge is not simply sending images to MathPix and displaying the results. The integration must be real-time, providing recognition results within milliseconds so that the experience feels instantaneous. It must be intelligent, knowing when to trigger recognition (after the student finishes writing, not in the middle of a stroke). And it must handle the back-and-forth of correction gracefully -- when a student erases part of their work and rewrites it, the system must update its interpretation accordingly without losing context.

We have also built a sophisticated correction layer between MathPix's raw output and the final interpretation presented to the student. This layer uses knowledge of the Israeli curriculum to resolve ambiguities in context. If a fourth-grader writes something that MathPix interprets as a complex integral, our correction layer recognizes that this is extremely unlikely and searches for a more plausible interpretation. Similarly, if a student is working on a specific exercise that expects a certain form of answer, the system can use this context to disambiguate recognition results. This curriculum-aware post-processing significantly improves practical accuracy for our specific use case.

4. The Smart Canvas Experience

The Smart Canvas is where all of Kedmathic's handwriting recognition technology comes together into a unified, student-facing experience. When a student opens the Smart Canvas, they see a clean, infinite workspace -- a digital equivalent of a sheet of paper with no edges. They can write anywhere, zoom in for detailed work or zoom out for a broader view, and organize their work spatially just as they would on physical paper. The canvas feels like paper, but it is infinitely more powerful.

As the student writes, the Smart Canvas provides real-time recognition feedback. A small preview appears near the written content showing the system's interpretation in clean, typeset mathematics. If the student writes "2x^2 + 3x - 5 = 0", they see the beautifully formatted equation appear alongside their handwriting, confirming that the system has understood them correctly. This immediate feedback loop means errors in recognition are caught and corrected instantly, before they can cause problems downstream.

The canvas supports the full range of mathematical notation that Israeli students encounter from fourth grade through the Bagrut. Basic arithmetic, fractions, decimals, algebraic expressions, equations and inequalities, functions, trigonometry, calculus -- all can be written naturally and recognized accurately. The system even handles the mixing of mathematical notation with Hebrew or English text, a common occurrence when students write word problems or annotate their solutions with explanations.

One of the most popular features of the Smart Canvas is camera scanning. Students can point their device's camera at a problem in their textbook, worksheet, or even a whiteboard, and the app will recognize the mathematical content and import it into the canvas. From there, the student can work on the problem directly in the app, with full access to Kedmathic's AI-powered hints and feedback. This feature bridges the gap between physical and digital learning, allowing students to use Kedmathic with any math content, not just exercises generated within the app.

The canvas also integrates deeply with Kedmathic's AI tutoring system. When a student is working on a problem and gets stuck, they can request a hint that appears directly on the canvas, positioned near the relevant part of their work. If the AI detects an error in the student's work -- say, a sign mistake in the third step of solving an equation -- it can highlight the specific location on the canvas where the error occurred, making the feedback concrete and immediately actionable. This tight integration between the writing surface and the tutoring engine creates a seamless experience where the technology supports the student's natural workflow rather than interrupting it.

Drawings and work sessions can be saved and revisited at any time. Students build up a library of their mathematical work, which they can browse, review, and continue at will. Teachers can also access student canvas submissions, allowing them to see not just the final answer but the entire problem-solving process -- every step, every correction, every moment of hesitation captured in the stroke data. This window into students' mathematical thinking is invaluable for identifying misconceptions and providing targeted guidance that addresses the root of misunderstandings rather than just their surface symptoms.

5. What's Coming Next

The field of mathematical handwriting recognition is advancing rapidly, and at Kedmathic we are actively working on several next-generation capabilities that will further enhance the Smart Canvas experience. Some of these technologies are already in beta testing, while others represent our longer-term research agenda. All share a common goal: making the interaction between students and mathematical content as natural, intuitive, and friction-free as possible.

One of our most exciting near-term developments is continuous recognition. Currently, the system recognizes handwriting when the student pauses or explicitly requests recognition. The next version of our engine will recognize content continuously as it is written, displaying the typeset interpretation in real time, character by character. This eliminates the small delay between writing and recognition, creating an experience that feels truly instantaneous. The technical challenge is immense -- the system must make accurate predictions about partially written symbols and update its interpretation fluidly as each stroke is completed -- but early testing has shown remarkable results.

We are also developing advanced diagram recognition capabilities. Mathematics often involves geometric figures, graphs, and other visual elements that go beyond symbolic notation. Our next-generation Smart Canvas will be able to recognize hand-drawn geometric shapes -- circles, triangles, coordinate axes, function graphs -- and convert them into precise, interactive mathematical objects. A roughly drawn parabola could be recognized as a quadratic function, with the system inferring the equation and allowing the student to manipulate it algebraically. This fusion of visual and symbolic mathematics mirrors how mathematicians actually think and work.

Multi-modal input is another frontier. We envision a Smart Canvas where students can seamlessly combine handwriting, typing, voice input, and camera scanning within a single workspace. A student might scan a textbook problem, write their initial approach by hand, type a calculation using the keyboard, and then dictate their reasoning aloud -- all within the same canvas session. Each input modality would be recognized and integrated into a unified mathematical representation. This flexibility accommodates different working styles and different types of mathematical activity.

On the recognition accuracy front, we are investing in personalized handwriting models. Rather than relying solely on a general-purpose recognition engine, the system will learn each individual student's handwriting style over time. After a few weeks of use, the recognition engine will have built a personalized model that dramatically improves accuracy for that specific student's writing. This is particularly important for students whose handwriting is unusual or difficult to read -- the system adapts to them rather than requiring them to adapt to the system.

Looking further ahead, we see the Smart Canvas evolving into a complete mathematical thinking environment. Imagine a workspace where the AI does not just recognize what you write, but actively participates in the mathematical exploration. It might notice that you are approaching a problem from a particular angle and suggest an alternative method. It could recognize that a diagram you are drawing contains a hidden relationship and highlight it for you. It could connect the specific problem you are working on to broader mathematical concepts, building understanding that extends far beyond the individual exercise. This vision of a collaborative mathematical workspace -- where human intuition and AI capability work together seamlessly -- is the future we are building toward at Kedmathic.