TinyPortal

Development => Support => Topic started by: PoML on July 26, 2023, 03:34:46 PM

Title: Using some unicode symbols result in error message when saving articles...?
Post by: PoML on July 26, 2023, 03:34:46 PM
I tried to create a tips & tricks article on my forum, and though it was a good idea to use unicode symbols for profile, questions etc.
By trial and error, I have found that some symbols seem to make tinyportal editor choke on saving, in particular:

🔐  (padlock)
⚙️  (cogwheel)
👦  (face)
📧  (email)
📱   (phone)

With these symbols, I always get a pretty bad looking error and being unable to save: 
"Database error!
Incorrect string value: '\xF0\x9F\x94\x90Fil: /customers/c/c/a/c5jnzrz36/webroots/r538174/medlemmer/forum/TinyPortal/Base.php
Linje: 285"

Strange then that putting it in a regular forum post like this seems to work flawlessly?
I tested also entering those characters using the view/edit code function, which resulted in my text being copied multiple times in, seeming like a real database problem.

PS:  tested now to replace those 4 characters with the corresponding HTML entities, like 📱...  That works fine, unless I try to edit it again as the editor has then made them back into the entities it chokes over...
Title: Re: Using some unicode symbols result in error message when saving articles...?
Post by: PoML on January 13, 2024, 07:52:23 PM
With the calendar block also having an issue with unicode letters - maybe this is a common bugfix for ver 3.0.0 ?
Title: Re: Using some unicode symbols result in error message when saving articles...?
Post by: @rjen on January 13, 2024, 08:05:06 PM
No clue. What is the bug you see in the calendar?
Title: Re: Using some unicode symbols result in error message when saving articles...?
Post by: PoML on January 15, 2024, 11:47:25 AM
In the calendar block, it creates the days but display of letter 'ø' does not work:
(link to report for that https://www.tinyportal.net/index.php?topic=36866.msg303510#msg303510  - I got a reply there, but the weekdays seem to be created from the locale rather than being a list)
Title: Re: Using some unicode symbols result in error message when saving articles...?
Post by: @rjen on January 15, 2024, 12:05:33 PM
I'll see what I can do
Title: Re: Using some unicode symbols result in error message when saving articles...?
Post by: @rjen on January 15, 2024, 08:36:14 PM
Okay, it seems that the mini calendar code somehow borks it up when it is set to show 2 characters per day..

If I set the day length in the blockcode to 3 it shows the characters correctly.
$day_name_length = 3;

2024-01-15_213142.jpg

but when it is set to 2 it doesn not;
$day_name_length = 2;

2024-01-15_213407.jpg

@tino this is a bit out of my comfort zone. It is happening in this code.
Any tips?

//if day_name_length is >3, the full name of the day will be printed
foreach($day_names as $d)
$calendar .= '<th class="smalltext" abbr="'.htmlentities($d).'" style="padding: 4px 2px;">'.htmlentities($day_name_length < 4 ? substr($d,0,$day_name_length) : $d).'</th>';

$calendar .= "</tr>\n<tr style=\"text-align:right;\">";
}
Title: Re: Using some unicode symbols result in error message when saving articles...?
Post by: @rjen on January 15, 2024, 09:02:37 PM
Okay, I got it...

in the blockcode, change this line

$calendar .= '<th class="smalltext" abbr="'.htmlentities($d).'" style="padding: 4px 2px;">'.htmlentities($day_name_length < 4 ? substr($d,0,$day_name_length) : $d).'</th>';
to
$calendar .= '<th class="smalltext" abbr="'.htmlentities($d).'" style="padding: 4px 2px;">'.htmlentities($day_name_length < 4 ? mb_substr($d,0,$day_name_length) : $d).'</th>';
Title: Re: Using some unicode symbols result in error message when saving articles...?
Post by: @rjen on January 15, 2024, 09:08:22 PM
@tino this raised another question: should we consider replacing the use of substr to mb_substr throughout the code base?
Title: Re: Using some unicode symbols result in error message when saving articles...?
Post by: tino on January 15, 2024, 11:31:41 PM
Quote from: @rjen on January 15, 2024, 09:08:22 PM@tino this raised another question: should we consider replacing the use of substr to mb_substr throughout the code base?

I think in Utils there is a substr that tries mb_ and if it doesn't exist used substr we should use that really as mb_ might not be installed
Title: Re: Using some unicode symbols result in error message when saving articles...?
Post by: PoML on January 16, 2024, 03:38:11 PM
Quote from: @rjen on January 15, 2024, 09:02:37 PMOkay, I got it...

Thanks!
That worked!

Back to the original topic - could the mb_substr be the cause of saving Articles with unicode symbols as well?
I tried saving an article with an 📧  (email) 📱   (phone) symbols, and got the same error:
Database Error
Incorrect string value: '\xF0\x9F\x94\x90&n...' for column `c5jnzrz36_db538176`.`smf_tp_articles`.`body` at row 1
File: /customers/c/c/a/c5jnzrz36/webroots/r538174/medlemmer/forum/TinyPortal/Base.php
Line: 285

(and to reply to my own question; no those seem unrelated)

Title: Re: Using some unicode symbols result in error message when saving articles...?
Post by: @rjen on January 16, 2024, 04:40:11 PM
Quote from: PoML on January 16, 2024, 03:38:11 PMBack to the original topic - could the mb_substr be the cause of saving Articles with unicode symbols as well?
I tried saving an article with an 📧  (email) 📱   (phone) symbols, and got the same error:
Database Error
Incorrect string value: '\xF0\x9F\x94\x90&n...' for column `c5jnzrz36_db538176`.`smf_tp_articles`.`body` at row 1
File: /customers/c/c/a/c5jnzrz36/webroots/r538174/medlemmer/forum/TinyPortal/Base.php
Line: 285

(and to reply to my own question; no those seem unrelated)

Yep complete different topic. It seems that it should work if your database collation is utf8mb4_general_ci, which it is probably not.
I read somewhere that SMF has some sort of trick to deal with it, which explains why it is not a problem in posts.

I have no clue what that trick is, so I cannot help with that...