Zompist Bboard Again

I have decided to write a Forthy dynamically-typed scripting language on top of zeptoforth, which I have dubbed zeptoscript. What little I have tested so far seems to be working... with the big exception of that half the time the GC is borked (and half the time it works perfectly)...

And now the GC works, and it is already quite functional (e.g. I can do things like declare and use data structures, manipulate strings on the heap, create and execute closures, and like).

For all Linux users here: a severe vulnerability has been found in xz, seemingly giving a backdoor into SSH servers. The recommendation is to upgrade as soon as possible.

bradrn wrote: ↑Sat Mar 30, 2024 5:53 am For all Linux users here: a severe vulnerability has been found in xz, seemingly giving a backdoor into SSH servers. The recommendation is to upgrade as soon as possible.

Damn. This is going to be a long week at work.

zeptoscript is coming along well, and is really fun to play with and add features to; now you can do things like
which gives:

I just realized how to implement object-orientation in zeptoscript how I would like to implement it. There would be no superclasses or subclasses, just classes implementing methods instantiated into objects, which are declared independent of any class. For the sake of simplicity there would be no metaclasses. Code for it would look something like:
Each object would consist behinds the scene as a cell sequence whose first element would point to its class and its subsequent elements would point to the class's elements. In turn, its class would be internally a byte sequence (so its contents are ignored by the garbage collector) organized into pairs of cells where the first cell in each pair is a method key (where 0 indicates that that entry is empty) and the second cell in each pair points to the code implemeting said method.

For compilation into flash, how it would work is that each implemented method would be placed (except for the method code itself) in a linked list in RAM (which is possible due to the freedom provided by zeptoscript w.r.t. memory management; I could not have easily done this with zeptoforth, hence the design of zeptoforth's OO layer), and would only be written out to flash once implementing all the methods is complete.

Also, each method would be given a global ID, which would be a positive integer for methods specified in flash and a negative integer for methods specified in RAM. The method table for each class would be a hash table with integral keys using these ID's which would contain a power-of-two entries (this is important because it avoids the need for a modulus when looking up methods).

For some reason, I cannot get into irc.sorcery.net. It keeps saying "Unknown host. Maybe you misspelled it?" even though it's the same host I've always used.

malloc wrote: ↑Mon Apr 08, 2024 8:58 pm For some reason, I cannot get into irc.sorcery.net. It keeps saying "Unknown host. Maybe you misspelled it?" even though it's the same host I've always used.

Maybe the site is down?

Okay, OO in zeptoscript is working and is nice to use; I have always been a fan of the idea of disconnecting methods from any kind of class hierarchy -- zeptoscript OO has no inheritance, BTW -- and I have accomplished that with zeptoscript OO. I had wanted to do this with zeptoforth OO, but the limitations of the zeptoforth memory management model rendered this practically impossible, hence why I opted for a single-inheritance, methods-belonging-to-classes model for it. (However, zeptoscript class members, not methods, are specifically tied to their classes, and raise exceptions if you try to use them on any other classes.)

alice wrote: ↑Tue Apr 09, 2024 2:35 pm

malloc wrote: ↑Mon Apr 08, 2024 8:58 pm For some reason, I cannot get into irc.sorcery.net. It keeps saying "Unknown host. Maybe you misspelled it?" even though it's the same host I've always used.

Maybe the site is down?

I tried pinging irc.sorcery.net today and had no problem.

Some zeptoscript updates:

• I have written a basic guide to programming in zeptoscript. Note that it assumes at least a basic level of knowledge of programming in Forth.
• I have added the capability to have "type" classes (not to be confused with Haskell type classes) to enable the creation of methods such as show, hash, and equal? (which I will mention later) that can be called on anything.
• I have added a pair of convenience words, #[ and ]# for constructing lists, e.g. #[ 1 2 4 8 16 32 64 128 256 ]# ' . iter-list outputs 1 2 4 8 16 32 64 128 256 ok.
• I have updated my show, hash, and equal? methods in zeptoscript so they can hand larger lists without exploding the stack (by testing their arguments for whether they are valid lists non-recursively, and if they are iterating over them rather than recursively descending into them, e.g. #[ 1 2 4 8 16 32 64 128 256 ]# show type outputs #[ 1 2 4 8 16 32 64 128 256 ]# ok.
• I have added code for compiling non-string literal constant byte strings, e.g. begin-const-bytes foo $80 c, $81 c, $82 c, $83 c, end-const-bytes foo ' . iter outputs 128 129 130 131 ok.
• I have added "collect" words for generating cell sequences, byte sequences, and lists, e.g. 0 16 [: dup 1+ swap ;] collectl-cells ' . iter outputs 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 ok.

Cool stuff, travis. How does doing all this in a Forth-style environment differ from, say, a more tradtional C-style one, besides the obvious? Is it easier or harder? Etc.

alice wrote: ↑Sun Apr 28, 2024 2:22 pm Cool stuff, travis. How does doing all this in a Forth-style environment differ from, say, a more tradtional C-style one, besides the obvious? Is it easier or harder? Etc.

It is easier, because I have much closer control over the whole execution and compiling environments, and I can effectively replace the underlying Forth environment, yet at the same time I can readily interface with the underlying Forth environment as I see fit (i.e. "foreign" words). Were I to do this in a traditional C-style environment, I would have to write a new compiler and runtime from scratch, and interfacing with C code from within the inner environment would be much more difficult.

I have implemented continuations (think call/cc in Scheme, or in this case, in zeptoscript) and have implemented cooperative multitasking and inter-task channels with them. (Note that zeptoforth, which zeptoscript is on top of, has preemptive multitasking but zeptoscript itself cannot run on multiple zeptoforth tasks for reasons.)

I don't really know enough about programming to follow this, but - wasn't the original point of zeptoforth to go back to the basics with a bare metal rather than higher level language?

Raphael wrote: ↑Tue Apr 30, 2024 3:15 am I don't really know enough about programming to follow this, but - wasn't the original point of zeptoforth to go back to the basics with a bare metal rather than higher level language?

I will admit to having the same query. To me, the attraction of Forths generally is their low-level–ness — I don’t see the point of a ‘scripting Forth’, and once it’s dynamically-typed I’m not sure I’d really consider it a Forth at all.

bradrn wrote: ↑Tue Apr 30, 2024 5:13 am

Raphael wrote: ↑Tue Apr 30, 2024 3:15 am I don't really know enough about programming to follow this, but - wasn't the original point of zeptoforth to go back to the basics with a bare metal rather than higher level language?

I will admit to having the same query. To me, the attraction of Forths generally is their low-level–ness — I don’t see the point of a ‘scripting Forth’, and once it’s dynamically-typed I’m not sure I’d really consider it a Forth at all.

The point of zeptoforth was to have a compiler and operating system on top of bare metal. The point of zeptoscript is to have something like Scheme implemented on top of that operating system.

I fixed the major bug I had encountered in zeptoscript, which really had nothing to do with call/cc or multitasker and everything to do with the garbage collector running when execution tokens are created, which had a bug where the top of the stack would be duplicated. Now the multitasker works beautifully now that this bug has been fixed.

Apparently my brain has nothing else to do than think about a message from nearly two years ago. Oh well.

Raphael wrote: ↑Wed Aug 03, 2022 1:12 pm So why not simply put a Blu-Ray quality video file on a DVD, instead of using a whole new different medium?

Ryusenshi wrote: ↑Wed Aug 03, 2022 1:48 pm Remember that Blu-Ray discs are already 15 years old. Video encoding formats have kept improving, to the point that now it's possible to encode a Full HD film into a file that could fit on a DVD – if you have the brand-new codec to read it, and if your computer is powerful enough to decode it in real-time.

I think what I wrote is mostly true, but this isn't the main factor.

Imagine it's 2004. With the widespread adoption of HD TVs, there is an opportunity to release a way of playing HD videos. You have managed to develop a video encoding that can fit an entire FullHD movie on a regular DVD: let's call this format "HiRes-DVD" (HD-DVD is already taken). You can produce them with existing DVD machines. The main problem will be selling them.

The need for new players

How are customers going to read these new HiRes-DVDs? Since the physical format is the same as old DVDs, a computer with a DVD drive can read them. On a general-purpose computer, users can download your new codec and read your videos. Success! But most people want to watch movies on their TV, with a DVD player.

And here's the rub. Players are much less versatile that computers: they are very specific machines, which can only read videos encoded in a format they already know. For DVD-Video, that means the standard MPEG-2 video format (cutting-edge technology from 1995!). This means existing DVD players will not be able to read a HiRes-DVD: they will be able to physically read the 1s and 0s on the disc, but for them, the data will be gibberish. Plus, most existing players output through a SCART or composite cable, which are not capable of carrying an HD signal.

If you want people to buy your HiRes-DVDs, you'll first have to sell them a new HiRes-DVD player anyway. This reduces the benefits of keeping the same physical format. You'll probably save money on development, since you can re-use existing drives instead of creating new ones; then again, the video encoding will be more complex so you may have to spend more on the decoding circuit.

HiRes-DVD cannot be "the same format" as existing DVDs. They will need their own ecosystem of discs and players.

Format wars and the luck factor

You weren't the only one who had the same idea, unfortunately: you will have competitors. This means you'll be involved in a format war.

Whether they're in video, audio, games, whatever, format wars are always messy, because ultimately, they're a chicken-and-egg problem. People don't want to buy a new player/console/etc if there are no titles available for it, and editors don't want to release titles in a new format if nobody has the player for it. As the developer of a new format, how do you break the circle? A successful launch is a priority: you first want to get several editors onboard, then release your new player with as many titles as possible to convince people to buy it. Then, if you're lucky, the public buys the player; more editors see an installed base and release more titles; this encourages more people to buy your player; the success snowballs and you're rich. On the other hand, a botched launch can doom your format, even if it's technically fine.

If we go back in time: DVDs were a success, but before they existed, it would have been possible to fit videos on a plain CD with lower quality. Why didn't such a format exist? Well, it did! There was a Video CD format, and it was a decent success in Asia. But it never took off in Europe or North America: people kept buying VHS cassettes, until the DVD arrived. Bad timing? No obvious improvement in video quality over VHS? Lack of protection against piracy, which discouraged editors? All of the above? Who knows.

Being technically good isn't enough. You also need luck.

About video quality

You said that the non-lawyer-approved files "keep Blu-ray quality". But... is that even true? They are in HD resolution, sure, but that doesn't guarantee they keep the full quality.

Video encodings exist because it's impractical to store every single pixel of every single frame: you'd need an entire hard drive to store a single movie. So encoding algorithms have to cheat. They will decide that some details aren't perceptible for a human eye, so they aren't worth keeping. They will detect that the background is the same for several seconds, so they will store it once and then only keep track of changes. The results may vary depending on what happens onscreen: an indoor drama with long static shots may need fewer bytes than an action flick with tons of explosions and cuts every two seconds, even if they have the same length.

Because of those factors, encoding programs can be adjusted. They can save space by degrading the level of detail somewhat: some movements may be simplified, some quick-moving objects may appear blurry, some compression artifacts may appear, etc. So you can fit a Full HD movie in a 4 GB file, but the quality may not be optimal.

Is this loss of quality acceptable? Well, it depends. For someone who disregarded their lawyer's advice, went to a shady site, and chose the file that would download faster? Probably yes. For someone who bought your brand-new HiRes-DVD player, plugged it into their brand-new FullHD TV, and popped a brand-new HiRes-DVD copy of their favorite movie? Probably no. (Unless their favorite movie is one continuous shot of two people talking while sitting on a couch. Hey, I'm not judging.)

Compared to a high-capacity disc format, your product risks being seen as the cheaper, lower-quality option.

Blu-Rays exist because Sony pushed for them

In 2005, there were two competing standards for HD discs, both with higher density than DVDs: Blu-ray developed by Sony, and HD DVD developed by Toshiba. Both constructors had their own motivations, but ultimately, Sony won, so I'll concentrate on them.

Back in 2000, Sony had a huge success with the PlayStation 2: part of its appeal was that DVDs were just starting to take off, and the console doubled as a DVD player. (I know that's how I convinced my parents to buy one!) So, Sony wanted to replicate this success with the PlayStation 3, which doubled as a player for their new Blu-ray format. They invested a lot of money into it, and even sold the console at a loss for a few years.

At first, results were mixed: sales of the PS3 lagged against its console competitors, the Wii and the Xbox 360; people who had just upgraded their collection from VHS to DVD were not eager to upgrade it again to Blu-ray. But Sony stuck to their guns, and they did manage to drive HD DVD out of the market. Blu-rays may not have survived, if they hadn't been supported by such a giant as Sony.

Bottom line

In the end, your idea is far from stupid. None of the reasons above totally preclude the existence of a format similar to our hypothetical HiRes-DVD: it could have existed, if someone had developed it, marketed it, and successfully launched it. Heck, it's possible that some manufacturers did work on something similar for a while, but eventually canned it.

You may notice that your idea is exactly what happened ten years later in 2016. 4K Blu-rays are actually the same physical storage format as regular Blu-rays, just with a more modern encoding.

Wow, Ryusenshi, thank you, that's very informative!